- 09:31:35Z on 0.216.0: '2 disk(s) evaluated, 0 alert(s)' — the count now
matches the 2 persisted records, closing the disagreement that exposed R-335.
- The 0.215.0 -> 0.216.0 redeploy replaced the container and the state file
came back with a changed_at written by the PREVIOUS version, so the new
container loaded the pre-restart record instead of re-baselining. Scenario L
observed on real hardware, not just through the production-path unit test.
- R-332 narrowed accordingly: what remains unproven is an already-ALERTED disk
not re-alerting after a restart.
Includes the two clean live cycles, the warning-vs-warn notification_log proof,
the 13 red-proof outcomes (A reported as a finding — the spec's mutation for it
is not a valid red-proof), and section 14 on R-335, the aliasing defect found
live in v0.215.0 and fixed in v0.216.0.
Found on live hardware two hours after the v0.215.0 deploy, by noticing the
release's own positive observable disagreed with its own persisted artefact:
the check logged '3 disk(s) evaluated' while disk-health-state.json held two
records. demo-hp's c11-scratch and felhom-backup are the same NVMe and share
a durable id, so one disk was walked twice per run.
Not cosmetic. The loop writes a disk's record before the next entry reads it,
so the second copy of an aliased disk consumed the FIRST copy's write as its
prior: the disk sustained against ITSELF and reached Hiba on a first sighting,
defeating truth-table row 6 — the rule that separates a one-hour benign
excursion from a false critical. It would also have emitted two identical
events for one drive. Latent on demo-hp only because all counters are zero.
Each diskKey is now evaluated once per run. Both entries stay marked seen so
neither looks like a disappeared disk, and the card still renders both rows —
the dedup is about state and alerts, not display.
Red-proof run and reverted: deleting the guard makes the first sighting emit
Kind:2 (Hiba-from-sectors) at 8 sectors.
- CHANGELOG leads with the severity fix and the live warning-vs-warn proof.
- CONTEXT records the settled decisions so they are not re-litigated: Hiba is
the label for predicted failure (no fourth word); sustain before count and
why; the provenance of 64/55/60; phase 2 owns the new SMART attributes
because they are a wire change under G-1; phase 1 state is one record per
disk, not a series.
- README documents the 14-row ladder, the persisted state, the hourly cadence
and the five message shapes.
- REUSE pins the severity wire contract on PushEvent — the defect's real home,
so the next typo'd severity is caught at the table rather than in production
— and records priorFor vs cardPriorFor, which differ by one observation and
make the chip disagree with the email if mixed up.
Part 4 was gated on a measurement. On demo-hp (Tier 0) the controller's real
/disks fetch — fetchDisks, the same path the check uses, not the 60s card
cache — costs min 0.805s / median 0.821s / max 0.841s over 10 calls, all HTTP
200, across 3 physical disk rows (2 distinct devices). Median is 6x under the
5s bar, so the <5s branch applies and the interval drops 6h -> 1h.
Why it matters: the one real failing drive's benign excursion lasted about ONE
HOUR and cleared completely. A 6-hourly sampler can land either side of an
excursion like that, see nothing, and then catch the terminal run half a day
late. The smartd history that produced the whole analysis sampled every 30
minutes and only just resolved the shape.
One check cannot distinguish a loaded state from a silent re-baseline — a
forgetful controller is also silent on its first check. It betrays itself on
the second, when the rebuilt prior makes the disk look newly sustained and it
alerts all over again. Caught while building the companion red-proof: with the
state load skipped, the single-check version still passed.
Three defects made the disk-health feature silent in exactly the case it
exists for. Evidence: felhom.eu documentation/audits/DIAG-smart-passed-trap-2026-08-14.md
1. SEVERITY (the one that changes whether anything arrives at all).
NotifyDiskHealthDegraded emitted severity "warn", which is NOT in the
hub's accepted set {info,warning,error,critical}. The hub coerced it to
"info" (hub/internal/api/handler.go) and severityNotifies dropped it
(hub/internal/notify/dispatcher.go), so every Figyelmeztetes-level disk
alert was filed as an informational notice and emailed to NOBODY, on the
customer and the operator leg alike. Now "warning". DiskAlertKind.Severity()
is exported so the contract is checkable from any package.
2. NO LEVEL ABOVE "worth an eye". smart_status.passed CANNOT fail on
unreadable sectors (attrs 187/197/198 all carry thresh 0 and a normalized
value floors at 1), so Hiba was unreachable for this whole fault class.
DiskVerdictFor now takes a DiskPrior and implements a 14-row top-down
ladder: sustained unreadable sectors, a count too large to be a blip (64),
unreadable+remapping together, overheating, NVMe critical flag or spent
endurance all reach Hiba. No fourth label — predicted failure is "Hiba".
3. IT SPOKE ONCE, AND FORGOT ON RESTART. The baseline was in-memory, so a box
that rebooted while a disk was failing never alerted again; and between 8
and 352 sectors nothing was emitted at all. State is now persisted
(disk-health-state.json, atomic tmp+rename), the decision compares against
the last ALERTED verdict (collapsing flaps to one alert while letting a
genuine escalation fire immediately), and a disk already at Hiba re-alerts
once it has BOTH doubled its count and waited out a 24h cooldown.
The card replays the same prior the check used (diskRecord.PriorSawUncorrectable)
so the chip and the email cannot disagree — the property the shared verdict
function exists to guarantee, now pinned rather than asserted.
Tests: 12 scenario groups A-L. Group L builds the Server through web.NewServer,
the same call main.go makes, over a real file.
MinAgent: 0.129.0
What was already right: the screen did not bluntly accuse. R-222/R-226 hedged,
naming both causes and the kept package, and saying it could not tell them apart.
That was honest - and it could not tell them apart because nothing ever looked.
Agent v0.129.0 looks, so the hedge becomes an answer.
New class RecoveryCodeOpensRetained on HTTP 422, gated by
FeatureRetainedRecoveryClass (MinAgent 0.129.0). The gate is SEPARATE from the
R-224 one because the two name different agent versions and a box can sit between
them, where a 422 is a shape we did not design. ClassifyRecoveryFailure therefore
takes both flags; the compiler found every call site.
The message says the code is correct, names the supersession date, says the
earlier package is kept, and says the CURRENT backups are unaffected - the half a
customer will otherwise assume wrong. It promises NO restore: there is no
in-product route to a set-aside store (R-312) and the retained package may itself
predate the repository-password field. It routes to support, which can do it.
The claim guard grew a surface and immediately convicted something. It scanned
templates only, while every recovery message is a Go string in a handler - the
highest-stakes copy in the product, never scanned. It now scans recovery_handlers.go
too, and found a PRE-EXISTING unregistered claim on its first run.
Six handler tests asserting which SENTENCE the customer sees; red-proofs asserted
applied, including: 422 unconditional makes an agent that never looked read as
having looked, and routing 400 to the new class congratulates a mistype.
The retrieval clause rendered unconditionally on every page and is false on a
reachable state - the same screen where the orphan card says we cannot tell.
The condition is a fingerprint PINNED at the decision, not a comparison against
the current key. The obvious proxy asks about the wrong key: the set-aside
copies were written under an older key the box no longer has, so on a
twice-rebuilt box the proxy promises about copies nothing can open. Demonstrated
- under the proxy, the replaced-package and legacy cases both flip back to
promising.
The pin is a recorded assumption and says so: nothing on the box records which
key wrote those copies. Empty is not a match. A countdown started before this
carries no pin and takes the cautious branch, not a backfill.
A sweep of all 36 templates found a fourth instance (backups page, same
condition applied) and a fifth (the confirmation screen, correctly left alone -
true at the moment of the decision).
New retrieval_promise_gate registers each claim with a reason rather than
banning a verb: a string ban failed twice, and the honest replacement copy
contains the stem.
The explanation paragraph - the always-visible half of the card - still ended
'a hozzajuk tartozo helyreallitasi koddal kesobb visszaallithatok lehetnek',
the same unevaluable claim v0.211.0 removed from the confirm block below it.
It survived because the spec called that line accurate, and because the
regression guard asserted the SINGULAR form while the card carried the plural,
which does not contain that substring. The guard now matches the stem, so any
conjugation fails it. The two accurate halves are kept.
Also: the guard's failure message sliced rendered HTML at a byte offset and cut
Hungarian mid-character; it now slices on rune boundaries.
The orphan card told a customer their set-aside off-site history may be
restorable later with their recovery code. The discriminator lives on the hub
and no wire field carries it, so the box rendering that card cannot evaluate the
promise. Copy replaced per the spec: state what happens, decline what we cannot
know and say why, name a route.
The claim page called the same three-word dashboard code two different names
depending on branch, one of which collides with the ten-word escrow code.
Retired 'Visszaallito kod'; the name is now constant and the sentence changes.
Naming only - a test pins that a reset code is still accepted.
secret_in_markup_gate no longer convicts Go template comments, which are
stripped before render; still convicts a real rendered secret.
After a reinstall the data drive could not be re-attached through any dashboard
route: both candidate lists came from the agent's unclaimed-disk scan, and the
rebuilt box's drives are claimed. The restore page said it was two clicks while
pointing at an empty picker.
The attach list now also carries the controller's own mounted-but-unregistered
filesystems. initialize is untouched, so the format wizard's system/backup
protection is unchanged. The 'two clicks' sentence is conditional on the picker
being non-empty, and says something true and actionable when it is not.
Both are one shape: something the box already knows, drawn as its opposite.
R-259 — A DISK WE FAILED TO READ WAS DRAWN AS A HEALTHY EMPTY DISK. readDiskUsage
(internal/system/info_linux.go) logged a statfs failure at DEBUG and returned, leaving the caller's
TotalGB/UsedGB/AvailGB/Percent at zero — and usageColor(0) is "nominal". The dashboard's
most-looked-at meter therefore rendered "0.0 GB / 0.0 GB (0%)" with a 0%-wide bar in the healthy
colour. "We could not look" and "there is plenty of room" were the same picture.
readDiskUsage now returns whether the measurement succeeded; SystemInfo gains DiskKnown and
HDDKnown (HDDConfigured is not a substitute: it says a path was configured, not that reading it
worked); and the template draws NO figure, NO percentage and NO meter fill when unknown, saying
"A tarhely merete most nem olvashato ki." instead. A healthy box is byte-identical, colour band
included.
This session rules the convention (felhom.eu CONTEXT.md S-39): an explicit `...Known bool` companion
beside the figures, checked in the template — the shape Offbox.StatsKnown already uses, whose own
comment says "a 0%-wide bar over an unread store is a picture of emptiness, and a picture is a
claim". Pointers and separate error fields are both legitimate Go, but a codebase with three
dialects cannot be gated (ROADMAP G-3 was blocked on exactly this). Existing call sites NOT
converted.
R-258 — THE PER-APP BACKUP TICK WAS GREEN ON PRESENCE, AND RED ONLY ON A GLOBAL CONDITION.
buildAppBackupRows set Tier1LastStatus from status.LastDBDump.Success, which is the box's single
most recent dump RUN, whichever app it belonged to. An app whose own dump failed showed a tick as
long as some other app dumped successfully afterwards; an app with no database took the nil branch
and went green on the mere existence of a restore point.
appDumpVerdict now reads THIS app's own entries in DBDumpStatus.Results (matched on
DumpResult.DB.StackName, failure = non-nil Error). Three states: any failing database -> error; all
clean -> ok; no result recorded -> NO verdict and no icon, titled "Errol a mentesrol nincs
eredmenyunk." The recovery unit carries no per-run outcome of its own, so green cannot honestly be
derived from presence. The global tier1DBStatus label is untouched — it is correct as a global.
RECENCY IS DELIBERATELY NOT ADDED. A tick over a three-week-old restore point is a real weakness,
but an age threshold means inventing a number and the time is already printed beside the icon.
Recorded as an observation, not changed.
AN EXISTING TEST WAS ASSERTING THE DEFECT AND WAS CORRECTED, NOT DELETED:
TestBuildAppBackupRows_Tier1FromRestorePoints expected "ok" for a status with no LastDBDump at all —
green from nothing but a file's existence. It now expects no verdict; its real subject, the
Tier1LastRun time, is unchanged.
The dashboard test EXTRACTS the meter block from the shipped template rather than copying it: a
copied block drifts, and a drifted copy passes while the page it claims to cover has changed — the
fixture-is-not-the-wire mistake this project has now hit twice.
Six red-proofs across both parts, each with the mutation asserted applied.
No new tag on any declared wire — report/builder.go maps into its own types and is untouched;
wire_contract_gate.py confirmed green.
go build / go vet / go test ./... green (28 packages), controller_gates --fast all OK, both run
separately from this commit.
The answer was on the wire and was discarded at the boundary, for the third time.
The hub has sent `escrow_stale` in the report ACK since v0.57.0 (json:"escrow_stale,omitempty").
report.EscrowStatus had no field for it, so encoding/json dropped it, and an empty restic_pw_sha256
had exactly one possible reading here: "hash-less supersession".
On demo-hp that reading was false in EVERY clause for four days, and the box told the customer so in
its own words. The hub HAD the hash and was withholding it because the escrow row carries a stale
flag (R-246); there had been no supersession; and the bundle DID cover the password — the hashes
matched exactly.
Fixed by receiving the field. EscrowStatus.Stale decodes, and reconcileEscrowed tells the two
conditions apart: a withheld hash now reports that the hub has flagged the row and is withholding,
that this box therefore cannot verify its bundle either way, and that it is NOT established that the
bundle fails to cover the password. The genuinely hash-less case keeps its wording.
Deliberately NOT changed, and said rather than skipped: the stale verdict itself (the hub's flag is
still the hub's verdict; runs still continue), and the customer-facing Hungarian card copy. Clearing
the wrong flag is an operator act hub-side (R-246); re-wording the card is UI work with its own
review path. This change is the wire and the diagnosis.
Found by felhom.eu/scripts/wire_contract_gate.py (G-1), which was built first and seen failing on 40
fields before anything was fixed, and which now refuses any new field of this shape.
go build / go vet / go test ./... green, run separately from this commit.
Records the deliverables, and is explicit about the limit on the live half: the
curl of an app info page could not be done, and names exactly what was tried —
crafty-controller is the only app declaring initial_credentials and is deployed
nowhere, and demo-hp's dashboard password in ~/.config/credentials no longer
authenticates (200 with no session cookie). A probe of the new routes was
discarded because its control killed it: real and bogus paths both 302 behind the
auth middleware.
§7.2's answer including the part that contradicts the task's premise: no line in
the repo says 'no silent auto-fill'; the rule is CONTEXT.md:2070 about accidental
EMPTY-password deployments. The hidden input is deliberate and untouched.
§7.4's measurement: the gate covers all 36 templates and catches a launder through
a local variable, but is blind to a secret under a neutral page-data key — the
exact shape of site two. Runtime coverage is 4 of 27 pages. Filed as R-255 rather
than described as complete.
§7.3: no evidence of actual exposure on the fleet, with the limit stated — it is a
current-state measurement and nothing recorded reads, which was part of the fault.
Also corrects v0.207.0's report: html/template STRIPS HTML comments; they do not
ship in the response body. Measured.
Site one. app_info.html rendered {{.InitialCreds.Password}} into a hidden span —
a REAL per-install credential, read live out of the running container, in the
response body of every render. The page now carries the non-secret half plus a
boolean; the value comes from POST /apps/<slug>/initial-credentials/reveal, which
RE-READS the container rather than serving a cached copy (caching it in the
handler would put it back in the body one layer in). no-store, CSRF-covered,
logged as an act. Both buttons go through it. A reveal that cannot read the value
SAYS SO rather than returning an empty string that renders as a blank password.
Site two, established before changing. The hidden input is NOT the defect and was
left alone: it fires only pre-deploy, and README §318 documents why the value must
round-trip — the customer notes the generated secrets down and submitting them
back is what makes the saved value the same one they saw. The defect was the
neighbouring READONLY input, which on an ALREADY-DEPLOYED app rendered the secret
into a page with nothing to submit. Fixed by POST /stacks/<name>/auto-field/reveal,
authorised by requiring a type:secret auto-field of that stack. Both directions
pinned.
The premise that this contradicted a repo rule does not hold: the rule is
CONTEXT.md:2070 'Password fields require explicit input — prevents accidental
empty-password deployments', about EMPTINESS. No line in the repo says 'no silent
auto-fill'.
The gate. scripts/secret_in_markup_gate.py, registered in controller_gates.py,
convicts any template expression that names a secret unless allowlisted with a
reason. Its limits are MEASURED and in its docstring: it catches a launder through
a local variable (the assignment names the secret) but is blind to a secret
arriving under a neutral page-data key — verified both ways. That is the shape of
site two, which this gate would NOT have caught. The runtime body assertion covers
all shapes but only 4 of 27 page templates; the other 23 are R-255, filed rather
than glossed. Two nets, different holes, both named.
Correction to v0.207.0's report: HTML comments do NOT ship in the response body
here — html/template strips them, text/template does not. Measured. A red-proof
planting a secret in a comment therefore correctly does not fail.
Records the deliverables: the raw response body before (1 occurrence, v0.206.0)
and after (0, v0.207.0) with a positive control in both directions; the §7.1
census finding two more instances of the render-then-hide pattern (R-254, one a
real per-install secret); §7.2's decision and why the promise was the wrong half;
every changed Hungarian string; all eight tests with their red-proof outcomes.
States plainly what was NOT proven live: R-252/R-253's notices could not be
rendered on VM 325 because both states are rebuild-only and the box re-registers
a drive on restart — the live run therefore exercised Scenario E instead, and the
notices are pinned at the template + predicate level with red-proofs.
Also records that red-proof D caught a fault in my own work: the explanatory HTML
comment quoted the old sentence, and HTML comments ship in the response body, so
the contradiction was still on the page and the assertion forbidding it could
never fail.
R-249. settings_security.html rendered the passphrase into a display:none
span behind a Megjelenit button. That toggle stops a browser DRAWING the value
and nothing else — the plaintext was in the response body of every render, so a
curl of the page returned it. Found by exactly that: it landed in a session
transcript while driving the documented rebuild path.
The codebase already stated this rule for the recovery code and this page did not
follow it (escrow_handlers.go: 'reveal (claim XHR only — R is NEVER templated
server-side into HTML)'). The page now carries only HasRetrievalPassword; the
value comes from POST /settings/retrieval-password/reveal — CSRF-covered because
POST, no-store, and LOGGED as an act, which reading it off the markup never was.
The tests assert the RAW RESPONSE BODY. Every test that asked what the customer
sees passed while the bytes carried the secret; that is why this survived.
Census: the render-then-hide pattern appears twice more — app_info.html (a real
per-install app password in a hidden span) and deploy.html. Filed as R-254, NOT
fixed here.
R-252. A rebuilt box keeps its drives but loses their REGISTRATION. The restore
page now states that before the customer presses anything, says the backups and
drives are both still there, and links to Tarhely > Meghajtok. Page and resolver
ask ONE question — HasRestoreDestination() reads the same
GetSchedulableStoragePaths() the scratch resolver reads.
R-253. The list promised 'a visszaallitas elobb ujratelepiti' three lines above a
refusal that fired BECAUSE the app was not installed. The promise was the wrong
half: reconstitution writes to the app's own GetStackHDDPath, which exists only
once the CUSTOMER has chosen a drive at deploy time. Auto-reinstalling would mean
the product making that choice for them. Copy now says to install first and routes
to /stacks/<app>/deploy.
Both notices are conditional — a healthy box renders as before, pinned by a test
that fails if either becomes unconditional.
Scenario A's live result first: on demo-hp in the rebuilt shape, no key was
minted on the real start-up offsite-apply path, and the hub received the state
it reports instead - offsite.state=awaiting_recovery_key with enabled:false.
Key restored byte-identical afterwards.
Includes Q4's seven rows mapped to the three states, the SEC 7.2 choice and
why, SEC 7.3's answer on the new-code button, every changed Hungarian string
quoted, all nine red-proofs with what was mutated, the R-245 reasoning, and
three observations noticed but not acted on.
CHANGELOG v0.206.0 with the ruling that reversed the fix, the three changes,
the SEC 7.2 staleness decision, Q7's closed trap, and the two bugs the tests
caught rather than review.
CONTEXT carries the three rules this session established, in the form the next
session needs them:
- a box does not create a repository key while the hub holds a sealed
package for it;
- the fact that answers a question must be kept where the question is asked;
- fix the state, do not remember that it is wrong.
REUSE gains four rows, each carrying the trap rather than just the signature:
the mint guard is a CONJUNCTION and t.Enabled is load-bearing in the derived
predicate; the discriminator ships INERT unless wired in main.go's confirmer
literal; the countdown removes BOTH halves or neither and must be driven by an
injected clock; and the epoch must be synced FIRST and unconditionally or the
falling edge is lost.
README documents the three customer-visible changes and the operator levers.
No version literal was edited: the controller version is ldflags-only.
REMINDERS (SEC 2.3). The offer epoch now stamps when it began, and the
undecided reminder escalates in EMPHASIS at 1, 3, 7 and 14 days.
THE READING IS STATED BECAUSE THE SPEC IS AMBIGUOUS, and it is written into
the code where it can be corrected. For an ABANDONING box, 5/3/1 are
unambiguously days REMAINING before a deletion. An undecided box has no
deadline - nothing counts down to anything, because SEC 7.5 deliberately does
NOT auto-abandon - so 14/7/3/1 cannot be "remaining" and are taken as days
ELAPSED, with the wording firming up rather than the bar appearing and
disappearing. If the operator meant something else, one function changes.
The stamp is re-set on every entry into the offered state, so a box that
settles and is later rebuilt starts its ladder again instead of inheriting an
old one.
OPERATOR LEVERS (SEC 7.5). --abandon-status, --abandon-extend=N and
--abandon-stop on the controller CLI, beside the existing operator
subcommands. They exist because the path that ACTUALLY happens is the customer
telephoning, and support needs something to press.
They live on the CLI and not in the customer UI deliberately: extending a
deletion the customer asked for is an operator judgement, and a customer who
wants it stopped already has the self-service route - they recover with their
code, which cancels it.
BOTH REFUSE RATHER THAN NO-OP, in two situations: when no countdown is
running, and when the store has already been deleted. A silent success is the
thing an operator most easily mistakes for "handled" - they would tell the
customer their data was safe when it is gone. Pinned by two tests.
--abandon-extend counts from NOW, not from the old due date, and a test proves
the old date passes without deleting anything.
Green: go build, go vet, go test ./... all pass; controller gates OK.
FULL PAGE ONCE PER ENTRY, NOT ONCE EVER. "Most nem" used to set a flag that
nothing ever cleared, so a box that abandoned its history and was rebuilt
months later - a genuinely NEW situation - would never see the page again. The
offer now carries an EPOCH, advanced on the edge into the offered state, and a
dismissal is recorded against the epoch it was made in. A fresh entry passes
the dismissal by arithmetic, with nothing to clear and nothing that can be
forgotten to clear.
That is NOT the flag the operator's ruling forbids. The forbidden thing
remembers that the customer decided so the screen can be suppressed while the
state stays wrong. This records WHICH SITUATION a dismissal was about.
A REAL BUG, caught by the test and not by review: the first draft returned
early from recoveryInterrupts when the offer was false, so the FALLING edge
was never recorded, RecoveryOfferActive stayed true through a settled period,
and the next entry counted as a continuation. The page never came back - the
exact defect the epoch exists to fix, reintroduced inside the fix. The sync is
now unconditional and the ordering is commented as load-bearing.
THREE LEVERS, THREE SCOPES, and none of them removes the route:
- clicking the bar away -> a browser SESSION cookie, cleared on login, so
the reminder is genuinely back at the next login. Nothing persisted.
- "ne emlekeztessen ujra" -> durable, epoch-scoped, silences the BANNER ONLY.
It starts no countdown, abandons nothing, and a fresh entry reminds again.
- "most nem" -> suppresses the full page only, as before.
The entry point on /backups/remote is bound to the OFFER and to nothing else,
pinned by a test that fires all three dismissals and asserts it survives.
SEC 7.3 / Q7 - THE TRAP DOES NOT SURVIVE THIS SESSION. While a recovery is
outstanding the "Helyrealitasi kod letrehozasa" button is UNAVAILABLE, not
merely captioned: creating a new code seals the current key, demotes the
package that opens the earlier history to retained custody that no shipped
path can read (R-199), and re-enables the recovery screen through the orphan
route while invalidating the code that screen accepts. A warning beside a
button is a warning people click past. The card now explains and points at
/recovery instead.
SEC 2.4 - the abandon confirmation changes with the behaviour. It used to
promise "felretesszuk - nem toroljuk". It now states the grace in days (from
the constant the countdown actually uses, never a literal in prose), that the
sealed package goes with it, that the customer can change their mind, where
the date is visible, and that the question does not come back afterwards.
The countdown is shown on /backups/remote for the WHOLE window - the bar
elsewhere is a nudge, this is the record, and a deletion date must be findable
on a quiet day too.
Tests: once-per-entry across a full settle-and-re-enter cycle; the banner
dismissal proven to be a session cookie (MaxAge 0, no Expires) and to persist
nothing; the opt-out proven to silence the banner while leaving the offer, the
route and the countdown untouched, and to remind again on a fresh entry; the
entry point surviving all three dismissals; a settled box showing nothing; and
the back-redirect refusing "//evil.example".
An existing test (TestRecovery_E) was updated: it asserted the legacy boolean,
which the epoch replaces. It now asserts the dismissal landed on the current
epoch, which is the stronger property.
Green: go build, go vet, go test ./... all pass; controller gates OK.
Until now "set aside" renamed the remote store and touched neither the escrow
nor the key, so the hub went on holding a sealed package for a key the box no
longer used. Shape (c) compares those two, finds them different, and offers
recovery - correctly, and for ever. A customer who had already said "I do not
want the old data" would be asked again at every login.
The operator's ruling is that the answer is NOT a "they decided" flag: fix the
state, do not remember that it is wrong. So the decision starts a countdown,
at the end of which the set-aside store and the sealed package that protects
it are removed TOGETHER. Afterwards shape (c) has nothing to compare and the
offer falls silent on its own - because the state is right, not because
something remembers it once was not.
THE GRACE IS REAL. The recovery offer stays reachable for the whole 14 days;
that is the change-of-mind path, and a grace in which recovery is impossible
would be decorative.
BOTH HALVES OR NEITHER. Removing only the store leaves a package that opens
nothing; removing only the package leaves ciphertext nobody can ever decrypt.
The two cannot be atomic across two machines, so it is a two-phase commit:
delete the store, record a durable marker, and keep DECLARING
offsite.abandon_purge_requested until the hub's ACK stops reporting a
superseded package. A crash between the halves re-declares on the next sweep;
it never leaves the pair half-removed and silent.
HUB HALF - SEC 8.2 ANSWERED: yes, the hub was needed, and only for this.
store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is
ever undone, and it never touches host_escrow (the package covering the key
the box uses now). The handler acts on the DECLARATION, never an inference,
and is placed immediately BEFORE the ACK is built - so
GetEscrowStatusForCustomer reads the effect and the SAME response closes the
box's two-phase commit. No second round-trip and no window where the box
thinks it is still owed. felhom-agent was NOT touched.
The countdown starts in ResetOrphanedRepo, NOT in the shared helper: the
helper is also the unclaimed auto-reset path, where nobody decided anything,
and an as-delivered box tidying a stranger's leftover store must not get a
customer's deletion clock. Pinned by a test.
Cancellation is wired into the recovery unlock, BEFORE the tier-up and the
listing - those can fail, and a countdown surviving a successful unlock
because a later step errored would delete the history the customer just
proved they can open.
The sweep is a Daily job at 05:10, not on the backup leg: it must run on a box
whose tier is not configured for runs. Quiet by construction on every box with
no countdown, and that silence is asserted.
Tests (all clock-injected; SEC 7.4 forbids shortening a live timer):
Scenario E (aside + package kept + countdown + offer still reachable, and
NOTHING deleted), Scenario F (both halves, the declaration repeating, the
close-out), Scenario G (cancel, path still nameable, no later deletion),
plus: not closed out while the package remains, a transport failure leaves the
countdown due and retrying, the no-op sweep issues zero remote commands, and
the unclaimed auto-reset starts no countdown.
RED-PROOFS, each with the mutation confirmed present in the file first:
F1) store deletion skipped -> Scenario F FAILS (no rm issued)
F2) declaration dropped from the report -> Scenario F FAILS (the hub is
never asked; the package would outlive the store for ever)
G) CancelAbandon made a no-op -> Scenario G FAILS (uncancellable countdown)
Green: controller and hub both build, vet and test clean; controller gates OK.
NOTHING WAS DELETED ANYWHERE - the terminal step has only ever run against
in-test fakes.
THE FACT WAS COMPUTED EVERY CYCLE AND KEPT NOWHERE. EscrowAutoConfirmer.Reconcile
has compared the hub's restic_pw_sha256 against the local key on every ACK since
SLICE 3. On the final-walk venue it logged, at 03:28:03Z and thirty-five minutes
before the customer looked, "the hub's escrow blob does not cover the CURRENT repo
password (hub hash 30ef574f != local 9b4a9a9d)" - and dropped it. The recovery
screen, evaluating in the same process, went on asking a question that could not
see it.
Now persisted: settings.HubEscrowKeySHA256 + HubEscrowKeyCheckedAt, recorded
UNCONDITIONALLY in Reconcile beside RecordPresence and RecordSuperseded - same
place, same reason: the box that needs it most is the rebuilt one with no target,
on which every gate below returns early.
OffsiteRecoveryOffer gains SHAPE (c): the hub holds a package for a key OTHER than
the one we are using. (a) and (b) are both proxies for that question and both have
now been wrong in opposite directions - (a) goes false the moment anything mints,
(b) is unreachable while the escrow is pending.
SEC 7.2, decided deliberately and stated in the code:
- a KNOWN DIFFERENCE offers, however old the reading. Age is not gated on. Both
sides are local; only the hub's half can be stale, and what the hub holds does
not change without a ceremony THIS box runs, which refreshes the hash on the
next ACK. Gating on age would make a box offline from the hub silently stop
offering - the exact failure this session removes. CheckedAt is persisted for
diagnosis, not as a gate.
- an ABSENT hash falls back to (a)/(b) and does NOT offer. "" is the hub
positively saying its package seals no repository password (legacy hash-less
escrow). Nothing to compare, and offering would put a permanent screen in
front of every legacy box.
The write damper: CheckedAt refreshes on every ack carrying a hash, but a save is
skipped when both the hash and the UTC day are unchanged, so an idle box does not
rewrite settings.json every fifteen minutes. It records WHEN WE LAST HEARD, not
when it last changed - the R-100 distinction.
Tests: Scenario C (a differing key offers, with both proxies asserted false first),
Scenario D (a matching key offers nothing), fact 1 still required, shape (a) still
works, and both SEC 7.2 halves.
RED-PROOFS, each with the mutation confirmed present in the file first:
D) hubHash != localHash conjunct dropped -> Scenario D FAILS (a healthy box
offered recovery forever); Scenario C still passes
WIRING) RecordEscrowKeyHash removed from the EscrowAutoConfirmer literal in
main.go -> TestMainWiresRecordEscrowKeyHash FAILS. This is the ships-inert
shape: unwired, everything compiles, every test in the package passes, the
auto-confirm still works, and shape (c) reads an empty hash forever.
Green: go build, go vet, go test ./... all pass.
THE DEFECT. WriteOffboxSecrets auto-generated on ONE input - does the file
exist. Its two neighbours in the same file, OffsiteRecoveryOffer and
needsOffsiteCredential, both consult GetHubEscrowIdentityPresent(). The same
fact was available on three paths and used on two.
Measured on the final walk: a rebuilt box's credential self-heal reached here
at 03:18:06Z and minted 9b4a9a9d over a hub package sealing 30ef574f. The
recovery screen then correctly reported nothing recoverable under the key the
box held. The screen was honest; the minting was not. And the flag was not
merely available at that moment - it was the PRECONDITION of the chain that
reached this function, logged at 02:48:03Z, six ticks earlier.
THE GUARD IS A CONJUNCTION, deliberately: a package held AND no key present.
A box the hub holds nothing for mints exactly as before.
The refusal is a HOLDING state, not a failure. ApplyOffsiteTarget catches the
sentinel and still writes the transport (ssh key, known_hosts, coordinates),
so the recovery screen can bring the tier up the instant the escrowed key is
placed (R-219). Returning the error instead would leave needsOffsiteCredential
true forever and the hub re-staging a consumed credential on every cycle.
New declared state offsite.state=awaiting_recovery_key, shown INERT to every
existing hub reader from their code rather than assumed: offsiteheal acts on
exactly one string; isStale needs Enabled && escrowed and this carries
Enabled=false; the delivery checker skips the applied shape; an unknown state
string is ignored by encoding/json. So NO hub change is needed for this part.
OffboxAwaitingRecoveryKey is DERIVED, not stored - the operator's ruling that
the state should be fixed rather than remembered, applied to this field too.
t.Enabled is load-bearing in that predicate and was MISSING in the first
draft. The existing TestOffsiteDeclare_DisabledTargetIsNotStranded caught it,
not review: a customer who switched off-site off is not awaiting anything.
Now pinned from the new predicate's own side as well.
Tests: Scenario A (no key written; transport still written; apply holds and
stages nothing), Scenario B (first-time box still mints), idempotency, the
nil-settings fail-safe, and the Scenario E carve-out.
RED-PROOFS, each with the mutation confirmed present in the file first:
A) guard block deleted -> both Scenario A tests FAIL with
"R-241 REGRESSION: apply minted a repository password over the sealed
package"; Scenario B still passes (the mutation is specific)
B) guard over-widened (hub-package conjunct dropped) -> Scenario B FAILS
with a first-time box unable to start; Scenario A still passes
Green: go build, go vet, go test ./... all pass; controller_gates all OK.
THE VERDICT. The R-203 block already said "a warning beside a success is read as a
success" and applied it to ONE of the two shapes it describes: an app missing a
declared mandatory FOLDER made the run incomplete, while an app skipped ENTIRELY
still reported ok. Both do now. Which skips count, decided by measurement:
selected+deployed with no recovery unit YES; selected but NOT deployed no (named,
with what to do — a box left amber by an app somebody removed is a status nobody
reads); disconnected/decommissioned drive no (own signal); nothing selected no.
LastSuccess and SnapshotCount still record what WAS captured.
THE FILED MECHANISM WAS NOT THE MEASURED CAUSE, and saying so is the point. §3
stated that toggling an app on leaves it without a bundle so the first run skips
it. Measured on demo-hp: the run's own pre-dump phase calls captureAllRecoveryUnits
for every DEPLOYED stack, through admitApp, before the push — a unit moved aside
was RECREATED and the run reported ok. That state does not survive a run.
What actually produced the 2026-08-06 sequence: the manual run was dropped by the
single-flight while an earlier run was still going. runOffboxBackup returned nil,
the handler had already answered "A tavoli mentes elindult", and the card then
showed the PREVIOUS run's green verdict — read as covering the app just selected.
The decision is now taken synchronously in the handler and a dropped request says
so. The nightly path still returns nil on purpose: nobody asked, and it retries.
§7.3 measured before deciding: CaptureRecoveryUnit writes a few KB of compose +
manifest, only ENUMERATES dumps rather than creating them, is idempotent and does
NOT stop the app — and already runs inside the off-site run. So there is no wait to
remove for a deployed app and NOTHING was built.
28 packages ok, 9/9 gates. Four red-proofs, each asserted to have applied. Fixture
note: the shared provider's ListDeployedStacks returned nil, so Scenario A first
passed for the wrong reason; fixed with an opt-in deployed set that defaults to nil.
R-237: /backups/restore listed apps that are CURRENTLY DEPLOYED and CURRENTLY
TOGGLED ON for future off-site backups. A rebuilt box has neither, so a household
that had just lost everything was shown nothing to restore while the repository
held their snapshots — measured live on the R-201 re-walk. To restore an app you
had to select it, to select it you had to have installed it, and to know what to
install you had to see the backup you could not see.
The store is now the source of the list (offsite_restore_list.go), built on the
existing R-193 OffsiteInventoryList. Installed-ness became a property OF a row,
never a filter on it. Every case is answered rather than hidden: a snapshot for an
app that is not installed is offered and says it will reinstall first; an installed
app with no snapshot is shown as having nothing; an unreadable store renders as
UNKNOWN (R-225's rule, one screen over) AND keeps the action, because "we could
not look" is not "there is nothing"; no-target is its own state. The felhom-offbox
and _shares marker tags are excluded from the app list.
R-238 classified as a HARNESS ARTIFACT: mode=full without confirm=1 is step 1 of a
deliberate two-step — it starts no job by design and redirects carrying
&full_prep=<app>, which deriveWizardStep requires to reveal the commit. A driver
that did not carry it forward landed back on the intent step. The operator's
browser run completed the same restore. The wizard's precedence rules were NOT
re-keyed: a stale ?full_prep= must never resurrect a commit button mid-restore.
The residue WAS real and is fixed: neither branch of that step wrote anything to
the log, so a refusal — including by the headroom gate — left no trace on the box.
Both branches now log, and so does the concurrent-op refusal.
resolveWizardApp is removed: it was dead once the gate moved, and its test pinned
the defect's behaviour (an untoggled app refused), which would have read as policy.
28 packages ok, 9/9 gates OK. Three red-proofs, each asserted to have applied.
Records the decisions rather than only the code:
- POLL not ACK, decided on Scenario B against the ACTUAL promises — the
no-target message gives no deadline and the card says 'within a day', so a
5-minute tick is inside both and no text needed changing. If either promise
tightens to minutes, go ACK-driven.
- The marker question: applied_marker lives in the guest's DataDir, which a
rebuild destroys, so it cannot suppress a legitimate re-run. Left alone.
- R-220 candidate (b), corroborated rather than a wider prefix, reading
/proc/mounts because the lsblk args are pinned in sudoers.
Live: Scenario C proven on demo-hp WITH a positive control — the job ran once
and logged nothing. A first reading counted 2 lines that turned out to be the
start-up reconcile, not the retry; the instrument was corrected before the
conclusion. Scenarios A and E are deliberately NOT live-proven here: both need a
rebuilt box, and that state arises naturally in Part 4.
R-218's declaration half shipped in v0.201.0 and works. Its consume half never
existed. Reconcile ran exactly twice per process — at start-up and when the
recovery screen drives it — and BOTH fire before the hub has anything staged,
because the hub stages in RESPONSE to the declaration those runs precede.
Measured on the R-201 re-walk: unlock reconcile 11:43:07, hub staged 11:44:57
saying 'next cycle', a full report cycle ran 11:55:46, still unconsumed at
12:06. A guest command line applied it in 18 seconds — everything correct except
the trigger.
Bridge.RetryIfDeclared re-runs the SAME reconcile on a 5-minute tick, driven from
the box's own published declaration (OffboxReportStatus().State) — the very
statement the hub acts on, so the two cannot disagree.
Poll, not an ACK flag, decided on the promise: the no-target message says 'amint
megvannak' (no deadline) and the card says 'within a day'. Five minutes is inside
both by a wide margin and needs no hub change.
It stops by construction — a healthy box does no work and logs nothing — and the
settle gate is deliberately kept via ReconcileWhenSettled.
The marker was investigated and left alone: applied_marker lives in the guest's
DataDir, which a rebuild destroys, so it cannot suppress a legitimate re-run.
R-220's customer half: the refusal no longer tells the customer to choose from a
list that may be empty. It names the rebuild, points at the Meghajtók page, and
promises no outcome.
Red-proofs: remove the retry -> credential uncollected (the dead end reproduced);
drop the stop condition -> a healthy box hammers the hub; call Reconcile instead
of ReconcileWhenSettled -> settle gate bypassed; restore the old sentence -> the
impossible action returns.
28 packages ok, vet clean, all controller gates OK.
Corrected in all four instruction files across all four repos. Found while confirming this
session own push by run ID, which is precisely the check that catches it.
In felhom-agent/CLAUDE.md the sentence contradicted the same file release section, which
already said R-168 mails the failure -- a contradiction inside one instruction file, the exact
class the R-229 work exists to find.
REPORT.md deliberately NOT overwritten in the sibling repos: a one-line docs correction must not
destroy the record of their last real implementation.
215 lines -> 110 (92 effective; block-level HTML comments are stripped before
injection and never reach the model, verified empirically on Claude Code 2.1.222
with a control and a treatment run).
Four new .claude/rules/*.md, each with a paths: glob list so it loads only when a
matching file is read: gates, ui-hungarian, backup-paths, agent-coupling.
The ## Layout tree was deleted as derivable; REUSE.md already owns the per-package
seams its annotations stood in for. The host/access table was deleted in favour of
a pointer to documentation/operations/nodes.md -- it carried three defects at once:
demo-felhom given as the LAN fallback address as if it were the route, a pinned
"agent 0.93.0" against the project's own no-versions-in-docs rule, and the claim
that no drill VM was provisioned on demo-hp. Measured live: qm list shows VM 300
drill-r50. felhom-agent/CLAUDE.md was right; this file was wrong.
Kept verbatim: the seven session-critical invariants, the F9 live-validation fence,
the end-of-session checklist.
controller_gates.py registers the shared instructions gate (felhom.eu/scripts/,
never copied here; an absent sibling clone FAILS).
Docs only -- no Go, no version bump, no image, no deploy.
Ledger: felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJc8sAGRWmavP3rMtdpkr2
CONTEXT gains the rule so it outlives the bug: on the unlock path the customer
is blamed only after a real attempt REFUSED their code; every other outcome,
including an unclassifiable one, says something else. Plus the two things that
must not be 'fixed' into it — elapsed time is never a classifier, and the error
TEXT is never read (when the distinction was not a value, the agent was changed
to provide one).
REPORT states the split honestly: the AGENT half is proven live on the venue
(400 -> 502 -> 400, same wrong code, only the hub's reachability changed), while
the controller's message selection rests on handler tests and red-proofs,
because /recovery correctly redirects since F7 set the old data aside and
restoring that state is the reconfiguration §11 forbids. Also records that the
correct codes were shredded by the previous session, so the live re-run used a
WRONG code — which makes the test harder, not weaker.
Two venue changes stated because they were not asked for, both restorations: a
fresh dashboard password (the previous session shredded it, leaving the box
impossible to log into) set through the supported --print-reset-code escape
hatch, and one normal off-site run to populate stats_known.
Declares the coupling: an agent below 0.126.0 answers 400 for both a fetch
failure and a wrong code, so FeatureRecoveryFailureClass withholds the refusal
reading and the 400 degrades to the neutral message. The gate blocks nothing —
it only decides whether the customer may be told to check their typing.
R-225 — an unread store said '0 pillanatkép / 0 / 50 GB' above a card stating
it held backups under another key. An SFTP listing found snapshot f3d9cd67 and
12 535 KB really there; snapshot_count and repo_size_bytes were simply ABSENT
and the zero value spoke for them. StatsKnown is now NAMED, for the same reason
OffsiteInventory.Empty is: zero is what an unread store and an empty one both
look like, and on the wire 'absent' and '0' are the same bytes. The fill bar
renders only when the fill is known — a 0%-wide bar is a picture of emptiness,
and a picture is a claim. A measured zero still says zero.
R-227 — WHICH LAYER ANSWERS: traefik, and this repo generates its config. But
traefik v3 serves no static files, so a branded proxy page needs a new always-up
container for every 502 on the box — out of proportion, and scoped in the report
rather than built. Shipped instead: the unlock posts via fetch and answers a
gateway failure in Hungarian without leaving the page. Progressive enhancement —
with no JS the plain POST is unchanged and still shows the proxy's error, which
the report says plainly rather than implying otherwise.
R-228 — the set-aside history was recorded in orphaned_renamed_to and read by
nobody: a census found zero references in any template or handler, while 12 535
KB sat at that path. It is surfaced as two facts and stops. It does NOT promise
the history can be reopened, because it cannot be by anyone today (R-199's
inventory is unbuilt) — and the set-aside CONFIRMATION copy was corrected for
the same reason: 'a helyreállítási kód nélkül többé nem lesznek megnyithatók'
implied that WITH the code they could be. The field's own comment called it
'recovery-code-recoverable', which was the same over-promise in the code.
Tests: scenarios F, G, H as render tests per branch of each gate. Red-proofs,
each demonstrated failing then restored: remove the StatsKnown guards (F,
'R-225 RETURNED: an unread store reports a snapshot COUNT of zero'), delete the
set-aside block (H). The F assertion on the fill bar is scoped to the bar's own
container — a bare width:0% search matched unrelated elements and would have
passed for the wrong reason.
28 packages ok, vet clean, all controller gates OK (the emoji gate caught a
warning sign in a template comment).
The failure branch was a two-way choice — superseded? M4 : M1 — and BOTH are
statements about the customer's code. rerr was never inspected, so a hub that
refused, an agent that was stopped and a genuinely mistyped code all produced
the same accusation. Measured live 2026-08-05 with a CORRECT current code: hub
firewalled off 0.0556s, agent stopped 0.0299s, against ~1.0s for a real unseal.
Five classes, from the VALUE and never the text:
hub-unreachable 502/503 from the agent — the code was NOT used
agent-unreachable no agent verdict at all (transport) — NOT used
no-bundle 404
bundle-too-old 409
asked-and-refused 400 — the ONLY class that may mention typing
unknown everything else -> NEUTRAL, the safe default
agentapi.RecoveryRefusal carries the status as a value (refusalError flattened
it into a sentence, and a sentence is not something a caller can branch on).
THE OLD-AGENT CASE IS WHY THIS NEEDS A COUPLING. Agent < 0.126.0 answers 400
for both a fetch failure and a wrong code, so a 400 from one cannot be read as
a refusal. FeatureRecoveryFailureClass (MinAgent 0.126.0) withholds that
reading and the 400 degrades to neutral. The gate BLOCKS NOTHING — it only
decides whether the customer may be told to check their typing.
R-226: the superseded message now names BOTH possibilities and restores the
ten-words prompt. The two are indistinguishable at the engine; the honest
message says so. It still does not promise the earlier package can be opened.
Elapsed time is logged (it is what diagnosed this) and is NEVER a classifier.
Tests: scenarios A-E at the HANDLER + the classifier table asserting the same
sentence under two statuses classifies two ways. Red-proofs, each demonstrated
failing then restored: delete the 502 case (A), remove the mistype clause (C),
default to the accusation (D), route an instant transport failure to the typing
message (E).
Two existing tests encoded the defect and were corrected, not deleted: the web
fake returned a BARE error for 'wrong code' (which is the shape of a failure we
cannot classify), and R-222's test forbade any mention of typing on a
superseded box — half of which R-226 deliberately reverses.
28 packages ok, vet clean, all controller gates OK.
R-216: the offsite key recovery is a coupled feature and now says so. featureProbes +
featureMinAgent 0.125.0 + a Supports gate at the unlock entry point, FAILING CLOSED — an
agent that cannot answer is named as such instead of the customer's code being blamed.
Measured live: a 404 from agent 0.120.0 came back as "we did not accept your recovery
code, check that all ten words", in 0.134 s, against a perfect code.
R-218: delete the repo-password short-circuit in needsOffsiteCredential. The declaration
stops when the TIER WORKS, not when a key exists — installing a key is the recovery
screen's whole job, so succeeding at recovery was switching off the mechanism that would
have delivered the coordinates to use it.
R-219: the unlock finishes the job — place the key, bring the tier up, then list. Without
it the promised listing could never render on the shape the screen exists for.
R-217: an unreadable store no longer claims to have opened with unattributable content
(the OffsiteInventory{} zero value). Opened / empty / unreadable are three states.
R-222: a code that is right about a RETAINED earlier package is named, not blamed. States
what the hub knows and promises nothing — no read path exists.
R-215: GET /recovery is gated on the same predicate as the interception.
Five red-proofs, each demonstrated failing and restored.
A customer whose machine was rebuilt had everything needed to get their data
back and no way to find out: the only route was a command line. This is the
screen that closes that.
IT UNLOCKS, AND ONLY UNLOCKS (operator ruling). It explains, takes the recovery
code, opens the repository and shows what is in there — apps, dates, sizes. It
restores nothing: restore is already per-app and lives in the backups area, and
a screen that unlocks and then offers to overwrite is two decisions wearing one
button.
ONE CORE, TWO CALLERS. RecoverInstallCore is split out of RecoverAndInstall; the
CLI wrapper keeps its exit codes and printed lines byte-identical, and the
handler drives the same function. Two implementations of the one operation that
can permanently lose a customer's data would drift, and only one would be
tested. Asserted from source on both sides by AST.
THREE WAYS OUT, none a dismiss button: recover; 'most nem' (the full page stops
interrupting, the backups-area entry point stays PERMANENTLY, bound to the offer
and never to the postpone flag); and 'I do not want the old data' — confirmed
TWICE and reaching the SHIPPED move-aside, which sets aside and never deletes.
THE CODE IS HANDLED NO MORE LOOSELY THAN ON THE COMMAND LINE: POST body only,
never logged, never persisted, never echoed, cleared on every path, no-store,
autocomplete off. No lockout — the code is a ten-word phrase, and locking a
customer out of their own data for a typo is worse than anything it prevents.
TWO DEFECTS THE TESTS CAUGHT, both fixed: an UNCLAIMED (legacy-open) box would
have been shown the page, because RequireAuth passes such a box through; and the
inventory nil-dereferenced when no off-site target was configured, which is
exactly the pristine rebuilt shape.
An absent off-site object has four meanings — never configured, mid-restart, a
transient config read failure, and rebuilt-and-stranded — and the hub cannot tell
them apart. The box can, from two local facts it holds with certainty, so it says
so instead of leaving the hub to deduce it from a silence (operator ruling).
The ACK's identity_blob_present is now recorded on EVERY ACK, before the gates
that used to discard it: on a box with no off-site target the auto-confirm returns
immediately, which is exactly a rebuilt box, so the one fact distinguishing it from
a box that never had off-site backups was thrown away every cycle.
The declaration needs BOTH halves — a fresh data area AND a hub-held recovery
package. Freshness alone is a box that never had off-site backups; dropping that
condition makes the whole fleet ask for credentials, which is what the Scenario B
test exists to catch.
The object carries enabled:false and zero sizes, which is what makes it inert to
the hub's existing fill and staleness checkers and to a pre-upgrade hub. A
configured box's JSON is byte-identical to v0.198.0's.
mode=unit restores the recovery unit — the app's definition, configuration
and database dumps — and NOT the customer's own files: RestoreOffboxScratch
passes --include <unit path> and the userdata in the same snapshot is excluded
by it. The outcome was one sentence for both modes and named neither scope,
so on the last step of a disaster recovery the customer was told the app had
been restored after the thing they were looking for had not been.
restoreScratchOutcomeMsg states what came back, what did not, and the next
step that gets it. The wizard's intent card states its scope before the choice.
The full-restore size gate is untouched and pinned as unchanged; the default
stays unit, since all three wizard forms set mode explicitly.
--print-reset-code runs as a separate process and persists the new code;
the running server's cache was never told, so the code the customer was told
to type was refused until the controller restarted. Nothing said so — during
the 2026-08-04 drill that cost two attempts with an operator present.
effectiveClaimCode now reads through to the persisted state before applying
the settings-vs-config precedence, which is itself unchanged. Read-through,
not a TTL: a TTL would leave a window in which a superseded code still works,
which is worse than the bug. Fails closed on an unreadable state; an absent
file is not an error.
The gap was already detected and warned about, in Hungarian, naming the app and the folders --
that warning is what stopped the R-201 drill. The defect was that the run still reported `ok`
beside it, and a warning standing beside a success is read as a success.
last_status gains "incomplete": minted, because "ok" | "error" | "running" had nothing meaning
"it ran, and this app is not fully protected". NOT "error" -- the rest of the run worked and
what was captured is real, so SnapshotCount and the LastSuccess anchor still record it. Half a
backup is not no backup.
The gaps are now recorded STRUCTURALLY (offboxRunResult.mandatoryGaps), not only as prose, so
the verdict has something to act on. It reaches the operator through the EXISTING per-run digest
(backup_run_failures) rather than a new event type -- a new type is a two-repo change and the
hub drops anything outside allowedEventTypes.
The stat-filter gains the ClassMandatory check Tier 2 already had. It is a NO-OP today
(TierOffsite admits mandatory only), so no customer-visible warning disappears -- demonstrated
by widening the tier filter alone and watching the check hold the line.
ANTICIPATED: calibre-web on demo-hp has exactly this gap, so its off-site status becomes
incomplete the moment this ships. That is correct and is the point.
Red-proofs: my first Scenario-C proof PASSED because the test only reached offboxCaptureSet
while the mutation lives in runOffboxInternal -- a mutation the test cannot observe is not a
red-proof, and the fix was the test. The run-level test now fails under both mutations
(unreachable gap recording; unconditional ok).
ExportDataMounts lives in delete.go, which reads as a destructive path. IT IS NOT: its single
production caller is the .fab export adapter, and nothing deletes based on its result. The
delete path's own guard, ProtectedHDDPaths, is layout-agnostic by construction -- it protects
BOTH <hdd>/... and <hdd>/felhom-data/... -- so deletion was never affected by the
namespace-root defect. That scope note is now in the function's doc comment, because the file
placement will mislead the next reader exactly as it misled the spec for this change.
Separated into its own commit anyway, so a change to a function whose filename says "delete"
is reviewable on its own.
An empty nsRoot falls back to hddPath -- the pre-R-203 shape -- so any caller not yet updated
keeps working on enrolled drives.
Tests cover both drive kinds and assert the NEGATIVE: no emitted path lies outside the app's
own data roots. Red-proof: leaving the site bare fails the system-drive row, emitting
/mnt/sys_drive/userdata where the canonical root is /mnt/sys_drive/felhom-data/userdata.
appbackup's path helpers take a NAMESPACE ROOT. Five call sites passed a bare DRIVE path.
On an enrolled drive the two coincide, so nothing showed; on the system-data fallback they
differ by exactly the felhom-data segment, and the app then bound a directory the off-site
capture set never looked at -- while the run reported ok. Measured live on demo-hp: the app
wrote to /mnt/sys_drive/userdata/media/books, the capture set looked for
/mnt/sys_drive/felhom-data/userdata/media/books.
THE RULE NOW HAS ONE EXPRESSION. appbackup.NamespaceRootFor / IsEnrolledDrive encode the
drive-kind comparison; backup.Manager.namespaceRoot and stacks.Manager.inGuest delegate to
it. There were already TWO copies and they differed -- the backup package's compared without
filepath.Clean, the stacks package's with it, so a trailing slash from config would have
flipped the mode in one and not the other.
Sites routed through it:
- stacks/deploy.go withPathVars -> ${USERDATA_PATH} (the live defect)
- appexport/fabplan.go + export.go (via a new provider method)
- web/handlers.go FileBrowser mounts (latent: the system drive is
deliberately never a registered StoragePath, so this is the identity today)
ComputeFabBuckets now receives the namespace root, which is what ComputeCaptureSet has always
received -- so the export's classified paths and the backup's capture set describe the same
directories by construction instead of by coincidence.
Tests are table-driven over BOTH drive kinds, because this survived by being invisible on the
kind that already worked. Red-proofs observed: restoring the bare-path call fails the
system-drive row with the two paths differing by /felhom-data; inverting the drive-kind
comparison fails every enrolled row.
--recover-offsite-install is the sibling of --recover-offsite-check: same fetch/unseal path
through the agent, same STDIN discipline for R, but it PLACES the recovered repository
password via InjectOffboxPassword so a rebuilt box reopens the history it inherited.
Doing this by hand would put the offsite DATA key through a terminal, a clipboard and shell
history. In-process the value goes agent -> this process -> the 0600 file and is rendered
nowhere.
The confirmation is a SECOND invocation: without --confirm-install it prints both hashes and
writes nothing, so the operator sees the comparison before any write is possible.
Three outcomes, named distinctly: installed (no local password -- the rebuilt-box shape),
unchanged (identical key already present, nothing written), refused (a DIFFERENT key present;
installing would clobber the key the current repository is encrypted under, and no force
option is offered). Exit 2 for the refusal, distinct from 1 for a failed step.
Red-proof: removing the confirmation gate makes the dry run write, failing the test. The
R-persistence test carries a positive control -- a planted copy is found, then removed and not
found -- because an absence check is worth only what its sensitivity is.
--recover-offsite-check is a docker exec diagnostic in the shape of --print-reset-code: it
reads the customer's recovery code from STDIN, asks the agent to fetch this host's sealed
bundle and open it, and reports whether the recovered key matches the one on disk BY SHA256.
Two hashes and a verdict; never a password, never R, never a blob.
R comes from stdin and not a flag because a flag value is visible in ps, in shell history, in
a container's command line and in any transcript of the session that ran it.
IT COMPARES; IT DOES NOT INSTALL. The recovered password is never written to
offbox/repo_password -- installing changes a live box on a path nobody has walked, and that
link is next session's, with the drill around it. A test asserts the data dir is byte-unchanged
after a check; its red-proof (adding the install call) fails it.
Exit codes: 0 match, 2 clean MISMATCH, 1 a step failed -- "it failed" and "it worked and
disagreed" must never share a status. A box with no local password reports distinctly: that is
the rebuilt-box shape, where the next step is to install rather than compare.
Nothing customer-reachable ships here: no card, no form, no preview.
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed
events reached the hub and TWO operator emails went out. The hub's operator
cooldown key is customerID:eventType(+tier) and that event carries `app` but no
`tier`, so the key held no app identifier: the first refused app took the hour's
slot and every other app's failure was discarded BEFORE anything was written
down, leaving no row on any channel.
The obvious fix — put `app` in the key — was ruled against: on a full disk it
produces one email per app, the volume problem wearing the correctness problem's
clothes.
internal/backup/runsummary.go: a per-run collector with exactly admissionSet's
lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the
end and only when something failed. A clean run emits nothing.
The per-app event stays and becomes the RECORD — the hub routes it record-only,
stored and logged every time, never competing for an email slot. The record and
the notification are now different things.
Deliberate skips (disconnected, decommissioned) are excluded: they have their
own alert, and a nightly email about an unplugged drive is one the operator
learns to ignore.
A manual run always reports: the digest carries a unique run_id the cooldown
cannot collapse. Someone pressing the button is actively trying to get a backup.
THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a
capture failure found between runs would be recorded and never notified — a new
silence introduced while closing one. That path emits a digest with NO run_id,
so the ordinary 1-hour cooldown caps it exactly as before while the mail now
lists every failing app instead of whichever was first.
A refusal is recorded ONCE, where the verdict is taken, not at the three legs
that consult it — R-181's contract is one verdict per app per run. Noting it per
leg listed one refused app three times and produced "2 of 1 apps failed". Found
by the digest's own test, not in review.
Silence is safe because the hub's deadline check raises expected_backup_missed
from report freshness, independently of any mail this box sends
(monitor/deadline.go:396,417). Confirmed, not assumed.
7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first
attempt — the AST test walked the backup package and not main.go; the test was
fixed and the mutation re-run rather than the pass recorded.
Found by v0.193.0's own live proof run. The estimate was printed fixed to two
decimal GiB, so every app under ~10 MB rendered as "estimated 0.00 GiB write" —
which reads as "no estimate was available" and is the opposite of what happened.
Observed live on demo-hp 08:59:46: opengist's real 178 KB estimate printed as
0.00 GiB.
Shipped in the same session because it is the same defect class R-181 is about:
a message an operator cannot rely on is worse than no message.
The arithmetic is unchanged and still in GiB — the reserve's own unit, so the
comparison against FloorFreeGiB reads directly. Only the rendering moved to
humanizeBytes. estimatedWriteGiB -> estimatedWriteBytes, with the GiB conversion
done once at the point of comparison.
B2's capture floor (v0.192.0) was consulted in exactly ONE place —
captureAllRecoveryUnits, which writes a few KB. The two legs that write the BULK
into the same backups/primary/<app> tree, the DB dump and the volume dump, ran
FIRST and unguarded. Measured live on demo-hp 2026-08-03 06:40:03: opengist's
volume dump wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then
refused the cheap write it had already lost the argument to. Its refusal message
claimed "the previous unit is untouched" — measured false: that app's tar had
gone 182,272 B -> 2,147,666,432 B under a stale manifest. Sixth entry in
CLAUDE.md's table of shipped guarantees the code did not provide.
Fix: ONE admission verdict per app per run (internal/backup/admission.go), taken
before that app's FIRST write and covering all three legs — they write under one
per-app root, which is why one verdict can honestly cover them.
- Lazy, at the app's first write, NOT once at run start: app A's dump can put app
B under the reserve, so a run-start verdict reads a disk that no longer exists.
- Remembered for the run, never re-decided between an app's own legs — that is
the split this closes. Reset per run.
- Placed ahead of DumpAppVolumesSafe, which stops the stack as its first act, so
a refused app is never bounced. After the volume-less check, which has no write.
- Exactly one operator alert per refused app per run.
- Leg order unchanged: volume dumps still precede the capture.
The floor is now SIZE-AWARE: it asks whether THIS app's write would cross the
reserve, not only whether the filesystem is already below it — which is how an
app was admitted at 96% and then allowed to write 2 GB. Estimate = the app's
previous .sql + .tar on disk. No history -> headroom-only, deliberately, and the
alert says so.
A container-based du per volume was MEASURED and rejected: 66 timed runs on
demo-hp guest 9201, median ~355 ms/volume (341-404) on volumes holding tens of
KB — container start-up, not the walk. Decisive on top: docker run needs the
writable layer, so it can fail under exactly the pressure the reserve handles.
The message was NOT weakened; the behaviour was moved so the wording became true.
It now also names which term bound. Every claim is checked against a sha256
fingerprint of the tree it describes, never against the log line.
Still refuses and never deletes: nothing here is generational.
11 new tests through the production functions. The DB leg cannot run without
Docker, so its gate is pinned by an AST walk of backup.go asserting admitApp
precedes DumpOne (strings.Contains is insufficient — a commented-out call still
contains the string). 4 red-proofs demonstrated failing then restored.
Ships BEFORE the disk-layout merge it exists for, and is harmless on a box
that never gets it. The mp1 partition was a BULKHEAD as well as a ceiling:
it kept a runaway capture from filling the space the container runtime
needs, because /var/lib/docker was a different filesystem. After the merge
it is the same one, and a full Docker data-root is a stopped box.
The floor sits in captureAllRecoveryUnits, checked BEFORE anything is
written: below the reserve, that ONE app's capture is refused, its previous
unit is left byte-identical, the R-158 alert fires with the space figures,
and the loop continues.
Two terms whichever binds first (97% used / 1 GiB free) in fillwatch's
shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer
is always warned before a refusal can happen — a floor that fires before
its own warning is a silent failure wearing a threshold.
Headroom, never unit size: a per-unit cap would be R-163 rebuilt inside one
volume. Refuses, never deletes: nothing here is generational, so pruning
could only destroy a different app's only local copy; pruneStalePrimaryDirs
is an orphan sweep, not retention, and must not be repurposed.
Tests 1184 -> 1191. One fixture strengthened mid-red-proof: the "old 20 G
ceiling is gone" test sat at exactly 20 GB and survived a literal
UsedGB > 20 cap — hollow. Now 120 GB, and the mutation fails it.
Earned during v0.191.1's own live validation. After the customer had been
warned, a restart produced ZERO fillwatch lines — equally consistent with
'ran and chose silence' and 'never ran'. Proving the checker was alive
needed a deliberate crossing into the critical band.
For an edge-triggered check the quiet run IS the healthy steady state, so
that ambiguity is permanent rather than rare. Check now logs a per-RUN
summary on every run, counting unreadable separately from healthy so a
drive that has quietly gone unreadable cannot read as 'all fine'.
Found live on 9201: neither sched.Daily nor sched.Every fires on
registration, so a box booting with a filesystem already over the line
would stay silent for up to 24h — the R-100 shape, and the same gap the
hub's own checkers avoid by leaving already-breached keys unseeded at init.
The watcher now runs once 90s after startup as well. Safe because the check
is edge-triggered against persisted state: an already-warned filesystem
stays silent. The delay lets mounts settle so a drive still returning reads
as unreadable and is skipped rather than warned about. Pinned by an AST
assertion — the schedule registration alone no longer satisfies the test.
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills.
It emits the PRE-EXISTING disk_warning/disk_critical pair, which was
allowlisted, copy'd, default-enabled and checkbox'd with no producer in any
repo — the sixth "built but never wired" instance here. Two threshold terms
(85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at
both ends of this fleet's size range. Edge-triggered on escalation only,
state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil
usage read is never a warning and never clears one. Per filesystem, never
per app. Daily 03:30, before the nightly app-data legs.
R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture
fails, loop continuing, carrying the target filesystem's used/free bytes.
Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed,
which is customer-enabled and would email the customer about a failure they
cannot act on. D-c overrides R-158's own proposal here.
R-174: the app-stop guard no longer starts apps onto MISSING drives — a
regression in v0.189.0 code, found by review and closed the same session.
SetStarter got the raw stack manager, whose StartStack has no drive gate,
and Recover runs at startup. R-171 one path over. bootDriveGate could not be
reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read
vars assigned after Recover runs), so holder #3 is extracted into a shared
driveStartGate with a test pinning the delegation. ErrStartRefused splits a
refusal from a failure: both keep the marker, only Failed alarms, because
routing a deliberate hold into NotifyBackupFailed is the same false alarm.
Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
Found by live validation on 9201, not by review. GetStacks() is the Manager's
in-memory map refreshed by the scheduler every 10s; sampling it every 5s without
refreshing means two identical samples can mean the cache did not update rather
than that the fleet settled. A container removed ~5s before the window closed was
still in the sampled fleet and the sweep logged 'no boot-orphaned apps' for an app
that had none. sampleBootFleet now refreshes first; a refresh error degrades
rather than aborting the window.
R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix
was written). Replacing isBootOrphan's container-count term with recorded intent
made a drive-gate-stopped app read as a boot orphan: the gate stops apps with
`compose down` (zero containers) and never touches desired_state, because it is
not the customer. Observed on 9201 with the drive held unmounted — the sweep
found and started it, burned both attempts, and handed it to the dead-app alarm.
The write hazard did not materialise (the unbound mountpoint is host-root-owned
and the guest is unprivileged) but that protection is accidental and untested.
New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not
start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive
already refuses this; the sweep bypassed it.
R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a
fleet docker was still restoring — three of six hard resets. Now a settle-then-
sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at
the end; ends on settled or a 50s budget, and the log says which. The budget is
50s because settle+budget+one retry must stay under the 90s dead-app grace — a
test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn
rather than the grace being widened to hide it.
Widening the window made two more holders reachable, so the one gate covers all
three: an absent drive, a quiesce, and an in-flight app-data operation — reusing
quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks().
R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way
table; absent keeps the old hasContainers behaviour exactly. Its comment argued
for the container count and was rewritten. presentStable is untouched. The two
gates' agreement is pinned from both sides against one fixture table.
27/27 packages green; 6 red-proofs observed FAIL then restored.
The box stops inferring the customer's intent from a container count and reads
what they actually asked for.
Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).
ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.
Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).
Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.
No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
Reports, does not refuse — pushes go straight to main with no pull request, so there is no
merge for a status check to stand at. The refusing half is .githooks/pre-push, which is
per-clone and --no-verify-able; this half notices when that was skipped.
CI reproduces the workspace's SIBLING LAYOUT on purpose: this repo's entry point invokes
the shared reuse_refs_check that lives in the felhom.eu clone next door and is never copied
here, and this repo's REUSE.md cites wgsync/reconciler.go, which lives in the hub. Without
the sibling the gate fails closed with 'gate is MISSING' — correctly, but for the wrong
reason. No uses: step anywhere; no version bump; nothing built or deployed.
README's Enforcement block now names controller_gates.py as THE entry point rather than
listing five of the seven gates individually. CHANGELOG records the allowlist entry, the
runner, the hook, the seam test and their red-proofs, and states the no-bump explicitly so
the omission reads as a decision.
A census of all thirteen gate scripts across the four felhom repos on 2026-08-02 found that
every check a CLAUDE.md names was passing and two of the four nobody is told to run were
failing. This repo had seven gates and CLAUDE.md named two; four more were reachable only via
a line in REUSE.md, and docker_run_volume_path_gate.py — RED at census time — through one line
in REUSE.md and nothing else.
controller_gates.py runs all seven plus reuse_refs_check on the repo root, streams each gate's
own output, and exits worst-wins non-zero. The shared reuse checker stays in felhom.eu/scripts/
and is invoked across the workspace — never copied here, which would recreate the drift it
detects; an absent sibling clone FAILS the gate and prints the path tried.
.githooks/pre-push runs it with --fast and refuses the push. Per-clone and --no-verify-able,
both stated in the hook itself; a manual run WARNS when the clone is unarmed.
test_controller_gates.py is a SEAM test — it asserts each member gate's own distinctive stdout,
not the runner's summary line. Red-proofed: an inert run_gate still prints 'all controller
gates OK' and exits 0, and turns the seam test red.
Tooling only: no Go change, no image build, no deploy, no version bump.
realVolumeSize mounts the NAMED VOLUME read-only into a throwaway alpine to du it
from a container view. No host path is involved — docker resolves the volume name
daemon-side — so it is the same safe class as the internal/backup/backup.go entry.
The gate was right to demand review; this diff is that review, on its own, because
burying an allowlist widening inside a feature commit is how an allowlist stops
meaning anything.
Tooling only: no Go change, no build, no deploy, no version bump.
The recovery unit on the customer's drive now carries the PORTABLE secret
class, so Tier-1/Tier-2 restore no longer depends on the whole-guest tier.
A customer needs the drive and nothing else.
Part 0's rulings overturned the brief's recommendation, on evidence:
- the data_key flag is untrustworthy (4+ encryption keys the catalog itself
labels as such are unflagged) -> R-127
- a DB password is not resettable in practice: POSTGRES_PASSWORD is ignored
once PGDATA is non-empty, so a regenerated value leaves the app unable to
authenticate against its own restored rows while the dump replay still
reports success (proven on a throwaway postgres:16-alpine)
Ruling (operator): type:secret travels, type:password never does, minus the
nonPortableSecrets code register. Plaintext -- withholding the internet-
reachable class is what licenses that, and the two are coupled.
Precedence: the UNIT WINS over the guest -- the unit's secrets were captured
in the same run as the dumps beside them, so they match the data being
restored. The fail-closed data-key gate is unchanged.
Secret values are never logged; the manifest records NAMES only.
This is D5's precondition and it is now met.
An app's namespace root IS its backup root: namespaceRoot returns a non-system
drive path as-is, so the recovery unit lands at <HDD_PATH>/backups/primary/<stack>/.
On a NAS that sits inside the share, which FileBrowser binds WHOLE — share root,
:rslave, download:true.
The bind was NOT narrowed, and establishing why inverted the fix. The share-root
:rslave bind is load-bearing (a 2026-07-22 probe proved an in-container access
through it wakes the idle automount trigger), and scoping is undefinable anyway:
apps on a share store at <share>/<app>, there is no userdata/ layer, and creating
one would write Felhom convention onto a customer's own NAS, which R-67 forbids.
So the browsing surface cannot be narrowed and the backup tree must never be
placed under it. Operator ruling: refuse the placement, keep the browse bind.
Tier 2 already refuses network targets for this reason (F-6C-1).
Nothing stranded: zero apps on network storage across all six hub customers
including Peti. R-67's browse capability is byte-identical.
FIVE surfaces, not the four the register named — settings.RefuseAsAppNamespace is
the single predicate. The deploy POST is the real boundary (it accepts any
caller-supplied HDD_PATH; DeployStack validates only os.Stat). Surface 4,
handleStorageDecommission mode=migrate, guarded only its SOURCE, so a whole
namespace could be decommissioned ONTO a NAS — that one is not in the register.
Fails closed: /mnt/felhom-drives holds both kinds, Kind exists only on a
registered path, so an unregistered path under that root refuses.
Supersedes README's "NAS backup locality — decision A" (v0.118.0).
9 tests, all non-effect (nil stackMgr, so a guard that misses panics rather than
passing). 4 red-proofs, each mutation asserted to have landed.
Suite rc=0, 27 packages, 0 FAIL. vet rc=0. Template + emoji gates OK.
Two defects E-2d found on a real box, fixed in this order deliberately: the
message is corrected BEFORE it is put on screen, because switching on a banner
that lies is worse than a silent one.
R-114 — the third state. resolveBackupTargetState had two outcomes: a disk
claims the target (healthy), or nothing does (degraded, "the backup is on the
system disk"). The state "configured, and its drive is gone" had no branch, so
it fell into the second and inherited its message AND its offer. Observed live
with the target detached: degraded:true, target:"felhom-backup" plus the
system-disk copy (false -- the backup was on a drive that had vanished) plus
offer_path naming that same vanished drive as the remedy.
New BackupTargetState.TargetAbsent discriminates. Degraded keeps its meaning
("is there a problem") so the wire contract is unchanged for every consumer;
TargetAbsent answers "which problem", because the two have opposite remedies --
attach any second drive, versus reconnect THAT one. Copy routed through
degradedMessageFor so one place still decides what a customer reads. The offer
is suppressed on the branch itself, NOT left to firstOfferableDrive's
Disconnected skip: that flag is set by the agent-side gate in another repo
(R-113), and this state must be correct independently of it.
R-112 — the state finally has a consumer. The endpoint was byte-correct and
nothing in the product ever asked for it: templates fetch 18 distinct
/api/storage/* endpoints and backup-target[/assign] were the only two with zero
references. Server-rendered on /backups now, following the existing
SingleCopyWarning banner pattern -- not a 19th JS fetch, because a banner that
needs JavaScript to appear is one more thing that can silently not happen.
backupTargetView returns nil for healthy and unknown so those render nothing at
all. The offer control POSTs to the existing assign endpoint behind the standard
inline confirm, never auto-submits, and surfaces restart_required honestly
instead of adding a self-restart.
Scenario E (the seam test) drives backupsHandler over httptest and asserts the
RENDERED HTML -- handler -> view -> resolver -> template. It deliberately does
not call the resolver and assert a string, which would prove the resolver that
was never broken. Deleting the one line that sets data["BackupTarget"]
reproduces the R-112 state and fails every render assertion.
Tests 326 -> 338 (+12) in internal/web; suite green (27 packages); both template
gates pass. Three red-proofs run and reverted, files byte-identical after.
MinAgent unchanged at 0.113.0: R-114 reads BackupTarget/MountPath/GuestPath/Role,
none of which R-113 altered (it changed BoundUnderParent, which this code does
not read). demo-hp on agent 0.113.0 is not held.
The absent copy is verbatim the hub's customerMessages["backup_target_absent"]
so the banner and the email tell one story -- filed as a two-repo drift risk,
not solved.
NOT LIVE-VALIDATED. Scenario C cannot occur on a healthy box; Session C proves it.
Registered as /api/backup-target inside ServeStorageAPI, which main.go mounts ONLY
at /api/storage/. Live result: endpoint not found, while every unit test passed --
the tests called the handlers directly and never travelled the mount. Caught by
the first live call, which is why the live call is part of the procedure.
Moved to /api/storage/backup-target[/assign]. A new test asserts the dispatcher
source contains both paths, so a handler nothing routes to fails the suite --
the seam-wiring rule applied to a route rather than a button.
The half that makes the rest work: a degraded backup target recorded only in
config is the silent-degradation pattern this arc has spent a week removing.
Part 3 -- POST /api/backup-target/assign moves the target via the agent's
POST /backup/target. It is the ONLY writer of the role: registration does not set
it, the drive-gate does not, no scheduler does. Declining is not calling it. The
agent returns restart_required rather than restarting itself, because restarting
with a backup in flight records a spurious tier failure for a backup that
actually succeeded (E-1 did exactly that).
Part 4 -- GET /api/backup-target returns the state and, when degraded, Hungarian
copy in FACT -> CONSEQUENCE -> REMEDY order, pinned by a test: a customer told
only the fact cannot act on it.
Healthy renders NOTHING -- no badge, no reassurance, no tonal change.
degradedMessageFor is the single decision point, so exactly one place could start
decorating a working box. Red-proofed: reassuring on the healthy branch fails
Scenario E.
UNKNOWN is not degraded: an unreachable or pre-R-82 agent means we could not ask,
which is not evidence of degradation (R-88 Part 2's class).
A HOLLOW TEST caught by its own red-proof: TestUnknownStateRendersNothing used
{Known:false} with Degraded left false, so it passed even with the !Known guard
deleted -- the second condition covered for it. Now {Known:false, Degraded:true},
which fails properly. Without the red-proof the test would have been decoration.
State is derived from the AGENT, never from our intent flag: on the two boxes
migrated by hand in E-1 the intent was never recorded while the drive really is
the target.
MinAgent: 0.113.0
Green gate: build + vet + test rc=0 (27 packages), run separately from this commit.
Caught before deploy by tracing a.Path back to its source, not by a failure. The
0.184.0 image is superseded and must not be shipped.
ReconcileDriveGates resolves the target as isTarget[a.Path], and a.Path is the
REGISTERED StoragePath -- for an external drive that is the GUEST path
/mnt/felhom-drives/<name>, not the agent's host MountPath (/mnt/<name>) that
/disks reports. driveTargetByPath keyed on MountPath alone, so the lookup never
matched: every absent drive, the target included, fell through to the generic
storage_disconnected.
The alarm would have looked wired, passed its own unit tests, shipped, and been
silently wrong on exactly the drive it exists for -- the same defect class E-2b
was opened to fix, one level down.
Now keyed under BOTH paths, mirroring planDriveGates which already registers
present[] under GuestPath and MountPath for the same reason.
Red-proof: MountPath-only keying fails with "the backup target is not resolvable
by its GUEST path -- the gate passes a.Path (the registered StoragePath), so the
backup-target branch would never fire".
Green gate: build + vet + test rc=0, run separately from this commit.
NotifyStorageDisconnected and NotifyStorageReconnected were defined and called
from NOWHERE. Registered in allowedEventTypes, in DefaultEnabledEvents, and given
a Hungarian message on the hub -- and never invoked. A drive going absent produced
apps stopped, a WARN log and a UI badge, then silence on every channel. Verified
against the gitignored-cmd/ trap with a positive control. Fifth instance of this
class, found by E-2 Phase 0 rather than by a failure.
A drive that is ONLY a backup target has no apps to stop, so it was silent twice.
ReconcileDriveGates now calls both halves. When the absent drive is the whole-guest
backup target it raises the more specific backup_target_absent (error) instead --
never both, since two mails for one event trains people to ignore the channel --
and recovers as backup_target_restored (info, the existing pairing-gated pattern;
severityNotifies NOT widened). The recovery mirrors the alarm's choice or the
operator cannot match them.
Which drive is the target comes from the AGENT (/disks backup_target, >= 0.112.0),
not from our StoragePath.BackupTarget: that is customer INTENT, and on the two
boxes migrated by hand in E-1 the intent was never recorded while the drive really
is the target. An older agent omits the field -> false -> generic alarm, never a
wrong one.
Before this an absent backup target had NO prompt signal: the tier stays DUE
(targetStoragePresent checks name presence, never reachability), so the only
evidence was its own failure at the next due cycle, up to ~24h away. The R-100
shape.
Tests observe the WIRE, not a mock, because the failure class is "nothing
arrives": a real Notifier posts to an httptest hub and the test asserts the event
type and severity that actually went out.
MinAgent: 0.112.0
Green gate: build + vet + test rc=0 (27 packages), run separately from this commit.
Foundation only, no version bump -- nothing customer-visible changes yet. The
offer UI (Part 3), the degraded banner (Part 4) and the controller half of the
absent-target signal (Part 5) are NOT here; they are tracked as E-2 in
OPEN-ITEMS.md so this cannot become a sixth "seam built but never wired". The
fifth was found by E-2's own Phase 0: NotifyStorageDisconnected is defined and
called nowhere, so a drive going absent emits no event at all today.
StoragePath gains BackupTarget bool -- the sibling role to Schedulable/
IsDefault/Kind. It is INTENT, not truth: the authority is the agent's
backup.local_backup_target, and truth is read from GET /backup/tiers. This
records what the customer ASSIGNED so the controller can render the state,
notice the drive going absent, and detect drift.
Invariants, each pinned by a test asserting the CONSEQUENCE not the mechanism:
- a drive NEVER acquires the role by appearing (registration does not set it)
- exactly one carrier; assigning moves rather than duplicates
- sticky: a new bigger/faster drive does not steal an assigned target
- an absent target STAYS assigned -- clearing on disconnect would be a silent
retarget by omission ("no target configured" instead of "drive missing")
- a network share is refused (R-108 risk model; the role is about LOCAL disk
failure)
Red-proof C: adding auto-elevation to AddStoragePath fails
TestRegisteringDrivesNeverAssignsTheBackupTarget with
registering drives assigned the backup target "/mnt/hdd_1"
Attributes may suggest and refuse the absurd, never select: demo-felhom's backup
drive is an external USB HDD and BOTH demo boxes report removable=0, so a
transport rule disqualifies the reference drive and a removable rule finds no
candidate at all.
Green gate: build + vet + test rc=0, run separately from this commit.
Both are the system reporting healthy while the customer is not, and both live in the same
status-derivation code. Neither is fixed by making the system quieter.
C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read
it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog
templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the
app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén."
— at the moment the customer pressed it because files were missing, with 156 MB of BookStack's
data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same
sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit).
Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and
NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that
the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the
Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm
copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror,
so the second local copy that exists for drive loss is unreachable by any customer action.
C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to
IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a
sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's
60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no
longer contradicts the alarm on the same screen. README's claim that faults "still surface as
restarting" was a wish with no test — corrected in place; it is the seventh such instance.
Six red-proofs observed, including the one that matters most: adding StateRestarting to
IsDownState fails the brief-restart test with "every deploy and update would page the operator".
go test ./... rc=0, 27 packages, run and read separately from this commit.
Tier2LastRun is the attempt clock and was rendered as 'Legutóbbi másolat' in the restore
confirm dialog. New LastSuccess + SuccessTracked anchor; tier2Update makes the three
rebuild sites safe by construction. F-DIAG: six distinct causes, target-aware redaction.
LastRun records an attempt, not a result. New OffboxTarget.LastSuccess, set only on the
success branch via the pure offboxAnchorAfterRun rule, carried to the hub as last_success.
Closes two silent-wipe sites (settings save, hub re-apply).
deadapp-check had no observable at default info level: its per-cycle line goes
through Scheduler.dbg(), gated on logging.level==debug, so on a default box it is
never PRODUCED (not merely filtered) and cannot reach the always-DEBUG ring. A
30s interval also puts it on the scheduler's quiet path. 'No alarms' was
therefore indistinguishable from 'the detector never ran' — which undermines
confidence in the F-CRIT-1 fix in the field.
A periodic summary, not a line per run: at 30s a per-run line is 2880 lines/day,
which is why the original author chose silence. Every 20th scan (~10 min) emits
one INFO with the scan count, apps evaluated and apps down. A test pins the
cadence so it cannot be widened into uselessness.
Also corrects the 'unquiesce guaranteed by defer' comment — fault 10 established
the guarantee is the crash marker plus Recover().
Six red-proofs with observed failure text, the bounds chosen for contention
(15m retry / 3h alarm) justified against the agent's own 120m PBS restore-test
ceiling and the 12m01s longest observed restore-test, and the live replay of
both findings on demo-hp with the hub DB as arbiter:
- failed restart alarmed 9s after grace expiry; banner shows (stopped)
- a deliberate user stop stayed silent through 9 dead-app scans
- 409 contention produced 0 operator emails (8 -> 8)
- a real transport failure produced 1 (8 -> 9)
Same box, same event type; the only difference is 409 vs a genuine error.
F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
1. restartAll returned nothing, so the failure was logged and dropped and no
caller could learn a customer's app had not come back. It now returns the
stacks that failed; both call sites record the outcome.
2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
means the user stopped it'). The quiesce loop stops stacks by the same
compose-down path, so a failed restart is also StateStopped and was
whitelisted into silence. Loop.FailedRestarts() is now the only thing that
lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).
F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.
Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).
Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
A witness in a _test.go file fires on go test and go vet but NOT on go build
alone — and a build-only step is exactly how the R-88 Part 2 near-miss would have
shipped. Moved beside the type it pins, and added one for AgentVersionReporter.
No defect found: quiesceBackend and *Client both satisfy their interfaces today.
No version bump, no deploy — compile-time only.
Records the valve-licence table, why a missing field means legacy rather than
unknown, and the near-miss: TieredBackend is runtime-asserted, so the signature
change built and vetted clean while the adapter silently stopped satisfying it.
MinAgent: 0.105.0. scheduledRunAllowed fired on any nil age; it now requires a
licence from valveLicensed, which grants it for AgeStateAbsent and for a LEGACY
agent, and refuses it for AgeStateUnknown. An unreadable storage no longer
masquerades as a first-ever backup and no longer quiesces apps outside the window.
A missing wire field means legacy, not unknown — deliberately. Treating it as
unknown would stop the valve firing on un-upgraded boxes and starve genuinely new
ones. Degrade logged once; unrecognised future values also map to legacy.
Caught in passing: TieredBackend is satisfied by a RUNTIME assertion, so the
signature change compiled and vetted clean while quiesceBackend silently stopped
satisfying it — which would have degraded every box to the single-tier path with
no error. Added a compile-time witness.
Also corrects the notifier comment that claimed operator-only came from a missing
customerMessages entry; enforcement is hub-side operatorOnlyEvents (hub 0.79.0).
Includes the cooldown keying finding (per-event-type only; fixed narrowly with a
tier suffix), the 180s grace window derivation, and the end-to-end event evidence
with its control: the new type 200s, a bogus type 400s, and notification_log shows
one operator row and zero customer rows.
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest
backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import),
wired by an init-only setter because main.go builds the notifier after the loop.
Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and
recovery rides recordSuccess's existing bool. Uses NEW operator-only event types;
reusing backup_failed would have emailed the customer in Hungarian about a backup
they cannot act on, since it has a customerMessages entry and is in live
enabled_events. Requires hub >= v0.78.0.
R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how
BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE,
consumed at the same single derivation point. 180s grace, derived from the deploy
flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app
that genuinely fails to come back still alarms.
The quiesce loop is silent by design when nothing is due, so 'no error lines' is
not evidence. The observable that is: every /backup/due makes the agent read that
tier's storage content, which lands in pveproxy/access.log. Both tiers polled
every 5 min, both HTTP 200, tick phase consistent with the deploy restart.
Includes the Phase 0 findings: the nil is agent-side (case a, a type boundary
that cannot represent 'unknown'), the four restarts were deliberate rather than
self-update, and the loop produced zero backup_failed events because quiesce
never imports notify (filed R-97).
internal/quiesce had no failure counter, no backoff and no breaker, and the driver
is a plain 5-minute ticker — so a tier that was due and kept failing stopped and
restarted every customer app stack every 5 minutes indefinitely. Live on
demo-felhom 2026-07-27: three cycles in eleven minutes against an unreachable PBS
tier; it ended only because PBS recovered.
The breaker gates the QUIESCE, not the backup — the harm was the outage taken to
attempt it, so backed-off tiers are dropped before any stack is stopped. Per
target (a broken offsite tier must not suppress a healthy local one), 15m→30m→
1h→2h→4h capped, reset on success, never permanent, never applied to TriggerNow,
and stillRunning is not a failure. State is in-memory on purpose: forgetting a
backoff costs one attempt; persisting one could outlive the fix.
Part 3 invariant recorded on scheduledRunAllowed — a missing value means UNKNOWN,
and only a positive 'never' may fire the safety valve. Fourth instance of the
class (hub v0.12.0, v0.73.0, R-81, R-88).
Part 2 (unknown != never) is NOT in this commit: the agent returns byte-identical
responses for 'read errored' and 'never backed up', so the controller cannot tell
them apart. That needs an agent wire change and is tracked separately.
The section read 'NOT deployed'. True when written (17:56 CEST), stale 26 min
later when I deployed (18:22 CEST) and did not update it. REPORT.md is defined
as 'overwrite with the most recent state', so an artifact this project validates
against was left asserting the opposite of reality.
Precision matters on the apparent contradiction: a 17:01 quiesce log 'before' a
17:56 commit is a TIMEZONE artifact (controller logs UTC, git commits CEST), not
evidence of a false claim. In one clock: REPORT 17:56 -> deploy 18:22 -> quiesce
19:01. Consistent. That host=CEST/controller=UTC mismatch is a recorded project
trap and caught me once earlier the same day.
Also records the live multi-tier quiesce evidence and what remains unproven.
The multi-tier path only engages against an agent serving /backup/tiers, and the
agent rollout was still mid-flight, so both production controllers remain on
v0.173.0. Names the three things that are therefore unproven live.
Operator ruling 2026-07-26: let the first backup run as long as needed; other
backups shouldn't start until finished.
A first FULL offsite snapshot runs for hours, far past max_quiesce. When that
bound elapses the app resumes (unchanged), but the loop then started the NEXT
tier while the first was still uploading. Now it breaks and defers the rest to
a later poll — vzdump still holds the guest lock, so the second start would be
refused by the agent (409, v0.99.0) or fail on the lock, and a failed backup
never satisfies a cadence, so the tier would retry into the same wall forever.
pollTier returns (phase, stillRunning, err). The app still resumes exactly once.
Red-proof observed and restored; full suite green.
MinAgent UNCHANGED — degrades gracefully against ANY older agent.
The agent gained per-target tiers in v0.97.0. The controller owns quiescing,
so the multi-tier schedule is reconciled here: every due tier is collected up
front and run inside ONE quiesce window (one stop, N sequential backups, one
resume). Two cycles on the weekly night would mean two app outages for one
night's work.
Dedup rule: local-only -> one quiesce; PBS-only -> one quiesce; BOTH due ->
ONE window with both backups inside; neither -> no quiesce.
- quiesce.TieredBackend + BackupTier + ErrTiersUnsupported (optional extension)
- agentapi: BackupTiers/BackupDueFor/StartBackupFor/BackupStatusFor;
targetQuery("") yields an EMPTY suffix so untargeted hits the pre-R-82 route
byte-for-byte
- Loop.resolveDueTiers = the dedup rule in one place, agent order preserved
- quiesceAndPollTiers + pollTier: app stays quiesced until the LAST tier
snapshots (resuming earlier loses app-consistency on the DR tier). Consequence
stated in the docs: both-due-night downtime = first tier's full backup + last
tier's snapshot, which is why tiers run fast-first.
- Manual 'Mentes most' covers EVERY tier, due-ness ignored.
- Window-gate safety valve now uses the OLDEST due tier, so a stale DR tier
cannot be starved by a fresher local one.
Capability detection: /backup/tiers 404 = pre-R-82 agent (the documented
route-probe mechanism). Not a featureProbes row on purpose — the loop needs the
tier LIST, not a yes/no. Degrade logged exactly once per process.
Tests +11, full suite green. Red-proofs #2 and #3 observed and restored.
Source: felhom.eu/documentation/audits/DIAG-agent-channel-2026-07-26.md
bootstrap.DetectEndpointDrift names a controller.yaml vs bootstrap.json
local_api.endpoint divergence -- one ERROR carrying BOTH values and BOTH paths,
its own event type local_api_endpoint_drift, and its own Hungarian banner shown
ABOVE the channel banner because drift is the cause and "agent unreachable" the
symptom. It writes NOTHING: reconciling from bootstrap.json would clobber a
correct controller.yaml on any half-provisioned or hand-repaired guest, so the
authority ruling is deferred to R-78. Fail-safe silent on absent/unparseable/
incomplete bootstrap and on an empty endpoint (ensureLocalAPI's fill-if-missing
path is untouched). Fingerprint compared as a BOOLEAN only; token never
compared, logged or exposed.
EffectiveProtected now gates samba on Enabled && UserSet, mirroring BOTH of
reconcileSambaAt's early returns, and the doc comment is corrected in the same
change -- it claimed "detection and deployment agree in both directions" while
citing only !smb.Enabled, an assertion that went false when !smb.UserSet was
added. Not over-suppressed: sharing on WITH a password and a dead container
still alarms.
Channel log: the debounce placeholder is stateUnconfirmed (rendered "unseeded")
instead of "up", so a born-down channel no longer logs "up->down" and orUnseeded
stops being dead code. Logging only -- the placeholder is still matched in the
re-arm condition, so F2 born-down alerting is byte-for-byte unchanged and all
nine pre-existing channelhealth tests pass.
Tests 951 -> 959, all green. Red-proofs A (both directions), E and F.
MinAgent unchanged; felhom-agent untouched.
Found on the demo-hp live leg: with import, import/paperless and import/calibre
in the carry-list, the derived skeleton RE-CREATES a per-drive drop-zone on every
drive forever — the dead lookalike the canonical root exists to remove, and one
that is never backed up (class: excluded).
Not a zero-removals violation: nothing deletes what an existing box has. Both
demo boxes' old drop-zones were verified to hold zero files before the change.
Found on the demo-felhom deploy leg: ensuring only <sysNS>/userdata/import left
its parent at 755 root:root, because EnsureUserdataDir MkdirAll's intermediates
at plain 0755 and chmods only the leaf. That made the system drive's userdata
root the one on the box outside the 2775/gid-1000 convention.
${IMPORT_PATH} = <system namespace root>/userdata/import — ONE drop-zone per box,
on the system drive, injected at BOTH compose-env builders with NO per-drive
fallback (unresolvable leaves it unset so compose fails loudly rather than
quietly building a second, dead drop-zone).
Third BindRoot (RootImport) + Import list in BackupSpec, extended through
ValidateBackupSpec/ClassifyBinds. Load-bearing: a stale `userdata: import/<app>`
entry against the moved bind would be a WHOLE-BLOCK reject, taking the app's
mandatory hdd classification with it.
Exhaustive-root audit: resolveAbs/structuralGuard/ComputeCaptureSet/
ComputeFabBuckets now take importRoot explicitly (an import bind resolved
against hddPath would name a directory on the wrong drive); unresolvable is
refused loudly into Skipped. GetImportRoot added to both provider interfaces.
Catalog-derived skeleton: UserdataSkeleton() -> UserdataSkeletonCarry() +
BuildUserdataSkeleton(), SORTED. The carry-list makes zero-removals true by
construction (`documents` is in no catalog app but on both boxes) and is the
fresh-box floor. The sort is not tidiness: the naive map-order derivation
measured 20 distinct outputs from 20 identical runs, which with fbNeedsRecreate
is a fleet-wide FileBrowser restart loop.
One authoritative compose parser: ParseComposeUserdataMounts now delegates to
ParseComposeClassifiableBinds. Import root excluded from per-app migration.
Surfaces: FileBrowser /srv/beolvasas source; app-page "Hova tegyem a fajlokat?"
with PathEscape deep links (never QueryEscape) and class-driven copy;
data_paths: annotation with the Fork-3 asymmetry; system-owned beolvasas SMB
share refused server-side at handler AND store, button omitted in template.
Caught on the way: the sharing template's row struct was function-local, so
adding {{if .System}} would have 500'd every share row. ShareRow is now
package-level and the render test uses the handler's own type.
Tests 915 -> 949, all green. MinAgent unchanged.
agentapi.SmartSummary.ModelName mirrors the agent's model_name; the card row label
prefers the device model over the raw name/UUID, falling back to Name(+hint) on an
old agent. Additive. Test + red-proof (drop fallback -> A4 fails).
/ now 302s to /launcher (ONE canonical URL per page — launcher body never served at /);
the Vezérlőpult keeps /dashboard + its nav slot (operator ruling, reversing v0.163.0).
Nav: Indítópult active on /launcher, Vezérlőpult href=/dashboard active there. Post-login
default (/) and the topbar logo (/) flow through the redirect; login target unchanged.
Repointed 2 dashboard-card tests /→/dashboard. Corrected the stale vacation/agent-DOWN
note in CLAUDE.md (agent is up at 192.168.0.162:8443).
Tests: 302 target+status, /dashboard 200, nav hrefs/active; red-proof verified.
gofmt -w across the controller tree (46 files) so gofmt -l is empty — disarms the
formatting landmine where a targeted edit + accidental gofmt -w swept ~46 unrelated
files. Pure formatting: whitespace + gofmt's optional-semicolon removal in reflowed
inline closures. One doc comment reworded ('' -> 'the empty string') to avoid gofmt's
Go-1.19 doc-comment typographic substitition ('' -> curly quote) muddying its meaning.
No build/vet/test behavior change.
The agent defaults SMART to UNKNOWN on non-physical targets, so they showed as
spurious 'Nincs adat' rows. isPhysicalDisk now excludes those types (card + check).
Test strengthened: a PBS/LVM fixture with UNKNOWN SMART must still be excluded.
Consumes the agent v0.94.0 smart payload (MinAgent floor unchanged; feature-detect
by presence). One pure verdict fn agentapi.DiskVerdictFor shared by the dashboard
card and the 6h check. Card via a 60s /disks TTL cache (anti-smartctl-storm);
unreachable agent -> Nincs adat, page never blocks. disk-health-check (6h) emits
disk_health_degraded on a degradation only vs an in-memory baseline (first run
silent, recovery/UNKNOWN never notify, multi-attr -> one event). No global banner
(deliberate). Pairs with the hub allowlist bump.
Tests: verdict table (>=90 red-proof), notifier emit, check first-run-silent
(red-proof), degradation-once, recovery-silent, UNKNOWN-excluded, FAILING-critical,
nil-smart card, TTL cache.
ONE setting (window start W) drives every nightly leg at fixed, never-stored
offsets: DB dump at W, tier-2 at W+60m, off-box at W+105m (wrap-safe). Precedence
settings > controller.yaml db_dump_schedule > 02:30.
- scheduler.UpdateDaily: retime a daily job at runtime (no restart) via a per-job
buffered resched chan + a select case in runDailyJob.
- new pure package internal/backupwindow (LegTimes/GateWindow/EffectiveWindow).
- quiesce disk-tier window gate: scheduled cycles run only inside [W+2h,W+6h) with a
safety valve (age>cadence+24h runs regardless); manual TriggerNow never gated.
Backend.Due now also returns the backup age (from the agent's own /backup/due).
- backup page: Mentési időablak card (time input + derived leg/gate rows); POST
/backups/window validates -> saves -> UpdateDaily x3 -> flash.
Tests: 5 groups, all red-proofed. Agent/cadence//backup/due untouched.
.sidebar-logo was left-aligned; add auto side-margins so the 140px logo centers in
the header (desktop sidebar + mobile drawer share the element). Adds TestSidebarLogo_Centered.
Viktor pushed the text-outlined logo.svg to felhom.eu main (be9edb4) — wordmark is
now 17 real path glyphs. Swapped FelhomLogoSVG to it; stripped Inkscape's leftover
empty <text/> shells + font-* style leftovers on the paths via lxml DOM pass (glyphs
untouched, no text-to-path by CC), dropped editor namedview. Cleaned FelhomFaviconSVG
vestigial <text>. Both constants now free of <text>/font-family; viewBoxes unchanged;
palette + gradients preserved. TestLogoSVG_NoLiveText committed green.
Mobile nav was broken — the <=768px block predated the v0.146.0 accordion and
flattened .nav-links into a horizontal overflow-x strip, clipping the accordion's
nested sub-lists. Replaced with a sticky top bar + off-canvas left drawer that
reuses the vertical sidebar (accordion untouched), plus a no-js static fallback.
Removed the sidebar customer-name span (kept on login). Added ?v={{.Version}}
cache-bust to logo/favicon URLs (Cloudflare 4h edge-cache; 0.126.1 failure mode).
Part 4 (outlined-logo constant swap) gated out per §3a: live felhom.eu main still
serves a logo.svg with live <text>/font-family; constants unchanged.
5 new tests via the real layout/CSS render; nav_accordion invariants unchanged.
Feature-detected navigator.share opens the OS share sheet with title+text+URL only
(no QR files: — narrow support drops the URL when given file+URL). Hidden unless
supported; Link másolása stays the universal fallback and catches the non-cancel
rejection; AbortError is silent. Template JS + tests only. 2 red-proofs verified red.
Mint a 160-bit capability URL (/s/<token>) serving a standalone read-only guest
launcher: same tiles, opens apps in new tabs, no account, no admin session.
Information only, zero control — every privilege stays behind each app's own auth.
- /s/ pre-auth pass-through (after the claim gate) + session-CSRF exemption; guest
password POST carries its own pre-auth HMAC CSRF.
- Constant-time token match; empty stored token = disabled = byte-identical mux 404.
- Optional per-share password: separate bcrypt hash + own attempt map; signed cookie
= HMAC(token|passwordHash) keyed with web.session_secret, so rotate/change invalidates.
- Guest labels ride the v0.164.0 ruling; never expose internal state vocabulary.
- Token redacted in logs (/s/<redacted>); never in CHANGELOG/REPORT/CONTEXT.
- Admin modal: copy-link, QR (go-qrcode), set/clear password, rotate, disable.
- Tests: Groups A-G (14) + 3 red-proofs verified red.
A UI stop (Leallitas -> compose down -> StateStopped) is the user's own
action, not a fault, and must not raise the deadapp banner OR the
app_start_failed event. Filter at the single fix-3 derivation point:
extract scanDeployedAppRunStates's pure core to classifyRunStates and
change the down predicate to IsDownState(st.State) && st.State !=
StateStopped. Suppresses StateStopped from both the banner dead-list and
the notifier Down-set at once.
Rests on two invariants (recorded at the seam, README, CONTEXT):
I1 StopStack = compose down => zero containers => StateStopped
I2 P2 census: all catalog services unless-stopped => faults never rest
at stopped (they surface as exited/degraded).
IsDownState unchanged; out-of-band 'compose stop' (containers remain ->
exited) still alerts. Tests +4 (notify 3->4, main 4->7), both red-proofs
verified. No template/funcmap/notifier/counter/copy change.
(A) The launcher monogram bled through every transparent white logo — it rendered
unconditionally under the img. Now hidden by default (.launch-mono display:none),
revealed only when the img chain fails (onerror adds .launch-tile--noimg).
(B) The /static/app-placeholder.svg default reached only app_list_row. The four
other sanctioned app-logo onerror chains now match the canonical SVG->PNG->placeholder
grammar: backups_apps (aligned row), stacks (infra->infra-logo else placeholder),
app_info hero (screenshots still vanish), deploy (keeps .LogoURL/.LogoPNGURL).
Template/CSS only; no handler/funcmap change. 5 tests + 2 red-proofs.
New /launcher page: a grid of large tappable tiles, one per openable deployed
app (subdomain presence is the single openability criterion, shared with the
Megnyitás button via the extracted Server.subdomainMap helper). Colored tiles
(deterministic slug color or .felhom.yml brand_color), white glyph/monogram
fallback, target=_blank links for operational apps, greyed unclickable tiles for
stopped ones. First sidebar item; / stays the Vezérlőpult.
Universal app placeholder: new AppPlaceholderSVG served at
/static/app-placeholder.svg, now the default FallbackIcon on app_list_row so a
logo-less app shows a placeholder instead of visibility:hidden. Brand mark is
never an app placeholder.
New Metadata.BrandColor; new funcmap tileColor/initial. 10 new test functions +
4 red-proofs. No agent coupling; MinAgent unchanged.
demo-felhom 9201 + demo-hp 9201 (G1 break-glass, secret shredded) both clean +
healthy on 0.162.0 with the settle-gate GO line. Honest B' accounting recorded:
the floor is in-memory (not persisted) so both above-floor boxes logged
"awaiting floor knowledge" then GOed ~10s later once the report ACK landed —
the report-ACK latency the 90s sub-bound is sized to, not a regression. The
zero-wait-when-floor-known invariant stays unit-proven (test E).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ
The day-0 race (DIAG-f10): a fresh box boots below the operator floor, the
apply-bridge consumes the single-use offsite password, then ~35s later the
managed auto-floor update replaces the container mid-install -> the new process
finds no installed key -> consume -> 404 -> offsite dead until an operator
Re-issue. Recurs on every onboarding whose ISO floor lags the managed floor.
Ordering-only fix (consume/install/persist internals + the 404-no-oracle
contract + the Consumer UNTOUCHED; R-71(b) rejected-by-design):
- New seam offsiteapply.SettleProvider.SettleState() + SettleFunc adapter over
the self-updater's own GetFloor()/IsUpdateRunning() (no second floor path).
- Bridge.AwaitSettle polls 10s BEFORE the 3-min Reconcile ctx: defers while an
update runs or the box is below the known floor; GOes at/above floor on the
first poll with zero added latency (B'). Bounds 90s floor sub-bound / 5min
overall, both GO+WARN (hub that can't serve a floor can't serve a consume ->
no burn risk; R-71c is the belt). ReconcileWhenSettled = gate then reconcile.
- main.go: bridge goroutine moved after the updater is built; wired only when an
updater exists (nil Settle = reconcile immediately, old behavior).
Finding: the floor is in-memory (report-ACK ~5-10s), NOT persisted -> unknown on
any restart until the first ACK; the 90s sub-bound is sized to that.
Tests (injectable clock, fake SettleState, recorded Consumer): A-E + nil-provider
+ cancelled-gate. Four red-proofs all observed FAIL then restored: gate removed /
updateRunning branch / floor sub-bound / overall bound. Deferral paths ship
unit-proven + red-proofed, NOT live-fired -- their precondition is now
structurally prevented by the v1.25.0 build gate. Layering: gate prevents, (a)
defers, (c) heals.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ
Network shares bind their share ROOT :rslave into FileBrowser — no
skeleton, no userdata scoping, nothing written toward the NAS. Gate is
the stub classifier (stub ⇒ excluded from mounts AND sources — an
exposed stub swallows uploads the real mount later shadows); idle autofs
is healthy and included. Drives byte-identical. Add/remove trigger the
debounced sync. Phase-0 probe on demo-hp: GO (in-container rslave access
wakes the idle trigger). Red-proofs A + B run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Leg A: „Hálózat" card on Beállítások → Rendszer — Helyi cím (LAN),
Hálózati név (only while Megosztás is enabled), Átjáró; live per render,
stored nowhere (S-5), „—" on unavailable.
Leg B: network section in the Debug system dump (interfaces/route/DNS/
lan_address), best-effort per item via the samba-netns door.
Leg C: NetBIOS trap named — Szerver field helper text + a purely lexical
hint on unreachable failures for single-label non-IP names.
Design note: all guest-net reads go through docker exec into the
host-networked felhom-samba container (stacks/guestnet.go, one seam) —
the controller's own netns is the docker bridge, so /proc/net/route etc.
would answer 172.x (the S-2 trap). Red-proofs: A2 gate-drop and C2
lexical-invert both failed as required.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Shipped in v0.158.0, caught live within the hour. /apps/<slug> returned 500
for EVERY app: html/template cannot call a pointer-receiver method on a
non-addressable value, and appDetailHandler passes Meta as a VALUE inside a
map[string]interface{}.
It compiled and every test passed because nothing rendered app_info - the
catalog tests used the funcmap route, which takes a value and works either
way. A template method call is only checked when the template runs.
Adds TestAppInfoRendersForEveryLifecycle with the handler's exact data shape.
The previous commit landed only the new test/badge files: a 'git stash' used
to compare REUSE.md ref-check output silently dropped the staged index, so
every modification to an existing file was left behind and that commit does
not build. This adds the metadata field, the predicates, the fail-closed
deploy gate, the catalog filter, the funcmap entries, the template edits and
the docs that those tests exercise.
The catalog knew only 'present' or 'gone', and 'gone' orphans every customer
already running the app. lifecycle: in .felhom.yml withdraws an app from new
installs without touching anyone running it.
Deploy gate is server-side and fail-closed, before any mutation, with the
ruled Hungarian refusal - hiding a button is not a gate. Unknown values fail
OPEN (available + one WARN), deliberately opposite, so a typo or a newer
catalog cannot pull a working app out of every customer's list.
Orphan detection never sees the field - a red-proof adds that filter and shows
the abandoned app immediately reading as an orphan.
Badge plumbing is generic (MetaBadge + meta_badge partial) so R-56's
difficulty labels drop in with no new markup.
A bare 'controller' matches directories too, so rg silently skipped
cmd/controller/main.go (false no-caller readings) and new files there needed
git add -f. /controller still ignores the built binary.
shouldRecreateOnBoot keyed on Deployed+drive-present alone. Deployed stays
true across a Stop, so a drive-backed app the customer switched off was
silently restarted on every guest reboot (proven live: immich).
Requires len(Containers)>0 as well - R-52's existing-Exited vs absent
distinction. A UI Stop is compose down and removes the containers; a guest
that went down under a running app leaves them. Container STATE is still
deliberately NOT a filter: that would miss a not-yet-restarted or stuck-Exited
app, which is the bug the boot-id path exists to fix.
Evidence sampled before any recreate - recreate's own StopStack erases it.
Honoured Stops counted and logged separately from no-live-bind skips.
R-51: aggregateState's mixed branch returned StateRunning ("partial"), so a stack whose
MAIN container was dead behind live helpers alerted on nothing — immich-server sat Exited
for 18 h, 100 % unreachable, no banner and no app_start_failed (audit F4). New
StateDegraded: a DOWN member whose docker restart policy is always/unless-stopped is a
fault (degraded, a down state); no/on-failure is a finished one-shot and stays benign; an
unreadable policy fails CLOSED. The unhealthy/restarting/paused/unknown exclusions are
byte-identical — folding unhealthy into down is the flapping fix-3 avoided.
R-52: new internal/bootrecon — one bounded start-once sweep at startup (2 attempts, 30 s
apart) for apps an interrupted boot left behind, inside the 90 s boot grace so a success
is silent and a failure still alerts. A zero-container stack is NEVER touched: the UI's
Stop is compose down, so a deliberate stop survives a reboot.
Both features carry a production-path wiring test (the v0.154.0 / v0.91.0 inert-seam
class). The main() assertion is an AST walk, not strings.Contains — the substring version
passed its own red-proof, because a commented-out call still contains the string.
Red-proofs run and restored: mix branch reverted -> "running" on the immich fixture;
boot hook commented out -> wiring test fails; zero-container gate dropped -> the
user-stopped app gets started.
NOTE: controller/cmd/controller/ is matched by .gitignore's `controller` entry, so new
files there need `git add -f` (and ripgrep silently skips main.go without --no-ignore).
Fixes a defect shipped in v0.154.0, found by the operator on the first live
click-through of the new wizard.
backup.Manager carries TWO running booleans. `running` (read by IsRunning) is the
concurrency single-flight, acquired inside the background goroutine — and
RestoreOffboxScratch never acquires it at all. `opRunning` (read by RestoreStatus) is the
display flag, set synchronously by BeginRestoreOp in the handler.
The wizard sourced OpRunning from IsRunning(), so for „Ellenőrzés" and the full-restore
preparation — its two most-used and longest actions, both streaming from restic — the
execution step was unreachable: the page offered all three intents with live buttons
while a restore was running, and the progress banner contradicted the phase strip on the
same screen. Pressing anything there would have been refused by the handler, which is the
exact "offering a control guaranteed to fail" dishonesty R-48 exists to remove.
Fix: restoreOpInFlight(st) behind a documented seam, fed by a SINGLE RestoreStatus() read
per render so the strip, the suppression decision and the running-op name cannot diverge.
Why the tests missed it: the Scenario-E table proved deriveWizardStep behaves correctly
GIVEN OpRunning=true, but nothing proved the handler ever computes true — hollow at
exactly that seam. TestRestoreOpInFlight_UsesDisplayFlagNotConcurrencyFlag now drives a
real Manager through BeginRestoreOp and asserts the render suppresses every form.
Red-proofed against the v0.154.0 shape.
Also: „Eredmény" was a dead label. The strip's highlight is now its own derived Phase,
separate from Step — a finished restore returns to the intent step (everything available
again) while the strip reads „Eredmény" and an outcome card shows the result. Bounded by
restoreResultWindow (10 min) so a stale result cannot look fresh, and bound to the app so
a finished bookstack restore does not light immich's page with bookstack's message. The
card survives a reload; the redirect flash does not.
No new agent coupling — MinAgent stays 0.90.0.
The offsite restore list rendered up to five inline forms per app row. Two of them —
„Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" and „Teljes visszaállítás
(fájlok + adatbázis)" — were sibling buttons whose difference is whether the customer's
data comes back at all. That mis-selection CAUSED the round-2 incident: an operator who
had read the source pressed the missing-only button and /backup/offbox/reconstitute was
never hit (DIAG-immich-restore-round2-2026-07-19, finding 1).
The rule this establishes: two adjacent controls whose difference is "your data comes
back" vs "your data cannot come back" must never be distinguishable only by layout.
Each row now carries ONE „Visszaállítás…" entry linking to a per-app wizard at
GET /backups/restore/app?name=<app>: three intent CARDS with consequence sentences, a
visible phase strip, danger styling plus the R-43 double-confirm carried over verbatim on
the destructive one, and the pair-honesty panel bound to real OffsiteScratchPair facts.
deriveWizardStep is pure over (op running, size-gate flash, scratch ready); the step is
never accepted from the request, and a running op outranks a stale ?full_prep= so no
commit button survives into a restore. While ANY op runs every mutation form is suppressed
server-side instead of being offered and refused.
No new mutation endpoint: every card posts to the pre-existing /backup/offbox/* handler
with the same field names and gates, and the page works with JavaScript disabled.
internal/{backup,appbackup,selfupdate} untouched. R-45's job registry stays its own item.
Fixes a latent bug found on the way: offboxRedirectTo hardcoded "?" when appending its
flash, which against the wizard's ?name=<app> target would have buried the flash inside
the name value.
No new agent coupling — MinAgent stays 0.90.0.
C6: operator deleted the photos, EMPTIED THE TRASH, then restored through
the customer UI. 40 files placed (vs 6 in the non-destructive run) — the
files were really gone and really came back. 11 assets active, no schema
drift, timeline confirmed. Full UI path, no endpoint shortcuts.
R-23(a): the STOP-2 floor save turned out to be exactly the trigger. Hub set
the floor at 18:56:27 CEST; the controller logged 'wait woke: generation=1'
at 16:56:27 UTC — the same second — with the out-of-cycle report 2s later.
The wake is logged at DEBUG only, so it is invisible in docker logs and
lives only in the debug ring.
R-23(a) is NOT complete: the floor was set to a version the box already ran,
so nothing restarted and the restart-single-fire assertion is unexercised.
STOP-2 applied by the operator in the right order: Day-0 Golden -> 0.153.0
(sha256 matches the bake), Agent/MinAgent 0.90.0 unchanged, global floor ->
v0.153.0 saved last. Build guest 9100 destroyed.
Agent 0.90.1's absence from the hub dropdown is CORRECT, not a defect: it
was never published (Gitea 404; 0.90.0 returns 206) and felhom-pve runs
0.90.0. The v0.90.1 source is committed at 9596d5a, so R-39 is a
build+publish away.
CORRECTION: my earlier claim that the stored controller password was stale
was WRONG. Values in ~/.config/credentials are single-quoted and my cut
extraction kept the quotes, sending a 15-char string instead of the 13-char
password. Re-tested stripped: 302 + session. The credential was always
correct.
Also records honestly that the reconstitute restored over an already-good
state, so C6 still needs the destructive empty-the-trash drill.
build-golden.sh v2.1.0 on felhom-pve from the vacation site. P3 (registry
reachability) passed before any mutation; the one deviation from the 0.146.0
recipe was an empty template cache, handled with pveam download (approved).
First golden carrying all FOUR infra images — the list came from
--print-infra-images on the 0.153.0 binary itself, so the historical
3-image fallback never fired and felhom-samba:1.1.0 is baked.
Upload HTTP 201; anonymous GET 200 with an exact sha256 byte-match
(15fdd191f3c660a6...); ranged request 206; the hub pod can fetch it.
GL-1 teardown: token via 0600 env file (never argv), shredded; leak grep 0;
build guest 9100 left stopped.
Bake log retained at 180:/mnt/5_hdd/felhom.eu/drill/bake-0.153.0.log.
Endpoint-level supervised run against snapshot 49e7cb46, the same one that
aborted in round 2. Controller log shows stop -> 'Starting stack immich
services only: [immich-postgres]' -> replay rc-0 in 20s -> full start.
No 'already exists'. Operation reported success (round 2: failure). immich's
own DatabaseService logged 'No schema drift detected' twice, where round 2
left it reporting drift. 11 assets active, 4/4 containers healthy, 231
public indexes.
Credentials were supplied file-to-file, never echoed, and shredded with both
cookie jars at the end of the run.
Closes R-47. No new agent coupling — MinAgent stays 0.90.0.
The replay needs a running DB container, so both restore paths started the
WHOLE stack first, giving the application a window to rebuild the very schema
objects the dump was about to create. Measured live on 2026-07-19 (H4,
DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds
before the dump's CREATE INDEX, the replay aborted "already exists" under
ON_ERROR_STOP=1, and immich reported schema drift. The data survived only
because pg_dump emits COPY before CREATE INDEX.
Both paths now open a DB-ONLY window: only the stack's database service(s)
come up, the dump is replayed with the app still down, and the full start
runs only after the replay exits 0. Fail-closed: a dump with no identifiable
DB service refuses BEFORE the first mutation. Every exit from the window
still does a best-effort full start, so a failed restore never leaves a box
with a database and no application.
New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line
scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage
with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty
list — argument-less `up -d` is a full start); RedeployFromEnv split into
PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's
RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden
`up -d` inside the old name is what carried the defect on the local path.
19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed
effects, replay-failure bring-up, parser decoys, empty-list refusal); three
companion red-proofs run and reverted. 23/23 packages green.
Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0.
The fab-upload GC test stat-ed the .part immediately after observing the slot
free, but expireIdleUpload unlinks AFTER releasing the mutex. Passed alone,
failed in the full package once this release's render tests made web heavier.
Not a production defect - a new upload mints a fresh random .part. The test now
waits for the outcome it asserts on the same deadline; red-proofed by removing
the unlink from production.
Capture on the box disproved the first theory: macOS DOES send a correct NBNS
query for <NAME><20> and nmbd DOES answer it correctly in 140us (flags 0x8580,
RCODE=0, right address) - macOS just never acts on it. NetBIOS there feeds
legacy browsing, not smb:// URL resolution, so the bare name can never work on
a Mac and nmbd was never the broken part.
felhom-samba 1.1.0 adds avahi + dbus, with avahi-daemon.conf and the _smb._tcp
service file templated from FELHOM_SERVER_NAME so a rename re-advertises. Both
daemons are non-fatal on failure - a discovery gap must not become an outage.
v0.151.0's card offered smb://<NAME> for Mac, which is exactly the dead form;
now smb://<NAME>.local. Windows keeps the flat \\<NAME>, which nmbd serves
correctly. Red-proofed both directions.
NOT claimed: Finder-sidebar discovery - published and answering on the wire,
but not observed working on the test Mac. Recorded OPEN.
TestRenderSambaCompose pinned the literal 1.0.0 tag, so an image bump read as a
renderer regression; now derives from SambaImage and asserts non-:latest.
S-1: /sharing/status coerced idle->running on the PHASE channel, so the first
poll of every steady-state page load reported a terminal job that never ran and
the client's repaint-reload fired ~1.2s apart, forever. The coercion's real duty
(liveness must never be contradicted) belongs to the 'running' LEVEL field
beside it, and is now pinned by its own regression test.
S-4 core: a terminal 'running' is served exactly once, so a REAL bring-up cannot
re-arm the reload on the page it just caused. failed/needs_password/in-flight
are never consumed. Unified async-job feedback stays the ROADMAP item.
S-2/S-5: new connect card with the Windows form, the Mac form and the direct
smb://<IP>, read from the SAMBA container's netns (the controller is on a docker
bridge and would answer 172.x). Derived per render, cached nowhere - the address
is a DHCP lease. Underivable => the line is omitted.
sharing.html's <script> block is byte-identical to v0.150.0. Red-proofed three
ways. 23/23 packages green.
F7/R-53: app_export.html built the app's public URL as '<sub>.{{$.CSRFToken}}',
so the "Megnyitás" link was wrong for every app with a subdomain and a session
CSRF token was written into a URL. Template now uses {{$.Domain}}, and
exportPageHandler supplies the key — it builds its own data map instead of
going through baseData, which is where every other page gets it. The page's
real CSRF path (csrfH() reading the meta tag) is correct and untouched.
The 7 red internal/backup tests are green again, with no behaviour change.
TestTier2V2_* / TestSharesTier2* all failed for one environmental reason:
Tier-2's off-drive guard asks system.SamePhysicalDevice (st_dev equality)
whether a target is really a second disk, and every t.TempDir() here shares one
filesystem — so the guard correctly refused the fixture's "two drives" and the
tests never reached their subject ("nincs másik fizikai meghajtó").
Seam in the package's existing style: a nil-defaulted Manager.samePhysicalDevice
field + sameDevice wrapper, seven call sites routed through it. Nil resolves to
system.SamePhysicalDevice, so production is byte-for-byte unchanged; only the two
fixtures inject a fake modelling one drive per directory subtree. No assertion
weakened, nothing skipped/renamed/deleted; all 7 mutation-proved.
Also: the ssh->pct-exec ASCII-grep and heredoc-credential traps are now in
CLAUDE.md's live-validation section.
The card claimed "Utolsó mentés: Még nem futott" on every box, forever, even
with dumps on disk and db_dump_completed events in the hub. dashboard.html
branches on {{if .BackupStatus}} and reads .Success/.LastRun, but
dashboardHandler never set that key — so the {{if}} arm was unreachable and
the "never ran" else-branch rendered unconditionally. The neighbouring
"Adatbázisok: N mentve" row kept working because it reads DBDumpStatus, which
was passed; that is the contradiction the audit caught on the live box.
Fix is the one-line pass-through the template always expected:
data["BackupStatus"] = fullStatus.LastDBDump. *DBDumpStatus nil/non-nil maps
onto the template's branch, so a fresh box still reads "Még nem futott" and no
zero-value timestamp is fabricated. No template change, no new view-model.
Tests drive the real handler through ServeHTTP so they bite on the handler
wiring, not the template alone. Red-proofed: deleting the assignment fails
TestDashboardBackupCard_ShowsLastRun.
Origin: felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md (F3)
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
offsiteRestoreRootFor as THE home for backups/offsite-restore (it was open-coded
in three places). Two patterns: the detached-job+poll shape, now noting that FIVE
of them exist and agree on nothing so a sixth should extend rather than clone;
and streaming subprocess progress, which encodes the two traps tonight found —
a source reporting nothing is normal, and progress may only update on unit
completion, so degrade rather than fake a percentage. Seams: sambaImgFn and
offboxStreamRunner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
README gains the async-job feedback section, the TOC entries it was missing
(SMB was also absent), and the three new endpoints. CONTEXT records the slice
and, more usefully, that three of its four versions exist because the cards were
watched against real runs — the failure modes were all invisible to unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Third real run, third thing only a live run could show. The per-app legs
finished in ~15 seconds; the remaining 40 of the 57-second run was the shares leg
and forget --prune, during which the card sat frozen on "calibre-web — 8 / 8
fájl". The same frozen-looking silence 4c exists to remove, relocated to the end.
Progress now carries a phase. The post-app stages announce themselves and the
app-scoped counters are cleared when a phase starts, so the last app's finished
numbers are never shown against work that is no longer about that app. Starting
the next app clears the phase again. Pinned by a test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The v0.147.1 file-count fallback fixed the incremental case but not the one the
demo box actually hits. Watching a second real run: bookstack reported clean byte
progress (100%, 154.0 MB, 7/7 files — the byte path works), while immich sat at
files_done 1 of 46, bytes_done 0, for 42 seconds.
restic 0.14 only counts a file into bytes_done/files_done when it COMPLETES, so
an app dominated by a single large archive (immich's ~430MB volume tar) freezes
both counters. No percentage can move in that window, so stop trying to fake one.
restic keeps reporting current_files and seconds_elapsed throughout. The card now
names the file being processed and the elapsed time: "1 / 46 fájl (430.2 MB) ·
feldolgozás alatt: immich_upload.tar · 42 mp". "Working on this file for 42
seconds" is a completely different message from "0%", and it is the honest one.
The last known current_files value persists across ticks that omit it (restic
does not send it every tick, and blanking the label every other second is its own
flicker); switching app clears it so one app's file is never shown against
another. Both pinned by tests, with the real 42-second status line shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Found by watching the v0.147.0 card during a real manual run on the demo box,
which is the only way this was going to surface: a 430MB immich push reported 0%
for 40+ seconds and then completed.
The parser was not broken. restic was genuinely reporting no transferred bytes —
on an incremental run where nothing changed, bytes_done is omitempty on restic's
side so it is not even in the JSON, and percent_done stays 0 for the whole run.
Confirmed against the real schema by capturing backup --dry-run --json from
restic 0.14.0 in the controller image rather than guessing; those captured lines
are now quoted verbatim in the type's doc comment.
Why it mattered: a byte-only bar is indistinguishable from a hang in the COMMON
case, which is precisely the silence 4c set out to remove. Shipping it would have
traded "no feedback" for "feedback that says 0% and looks stuck".
files_done/total_files are now parsed and published alongside the bytes; the card
prefers bytes when bytes move, otherwise drives the bar from files and says
"N / M fájl ellenőrizve". parseResticStatus returns a struct instead of four
positional values, and a new test pins the real incremental line shape (bytes
absent, files climbing) so a refactor cannot quietly restore the stuck bar.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The systemic complaint, twice in one evening: you press a button and nothing
happens. No progress, no ETA, no named result. Three worst offenders, fixed on
the two patterns already here (deploy 3-step panel, storage-init status poll).
No new framework — that is a ROADMAP item; three targeted cards ship tonight.
4a — a verification restore names its result. The flash said the app had been
restored "to a verification folder on the drive"; which folder, on which drive,
was invisible, so the customer could not go and look at what they had just asked
for. Full path now. The restore page gained a listing of existing verification
copies (app, size, date, path) — nothing anywhere showed these, so they piled up
and the only way to find them was SSH — each with a double-confirmed delete.
That delete is the only one this release adds, so it names a STACK, never a
path: the Manager resolves the name inside a backups/offsite-restore root it
computed itself and refuses anything landing outside. Red-proofed — neutralise
the name guard and stack:"" resolves to the offsite-restore ROOT and takes every
copy with it. Refusals are asserted as non-effects.
4b — Megosztás enable shows what it is waiting for. Enabling ran ReconcileSamba
synchronously inside the POST handler; on a golden without felhom-samba baked
that is compose pulling ~100MB, i.e. minutes of an apparently-hung form post
followed by "Beállítás mentve." whether or not anything came up. Detached +
polled now, distinguishing "képfájl letöltése" from "indítás" — decided BEFORE
the work starts, since afterwards the image is always present. Success is
probed, not inferred (compose up -d exits 0 on a crash-loop). The password form
starts the same job: with UserSet false reconcile deploys nothing, so on a fresh
box that is where the pull actually happens.
4c — "Távoli mentés most" streams real progress. restic was already reporting
bytes and percent; the runner seam used CombinedOutput() and discarded them. The
manual run now passes --json and scans stdout line-by-line: total bytes, percent,
current app. Manual only — the nightly stays silent, pinned by a test that fails
if it ever passes --json. The poll now arms unconditionally, closing a race the
manual trigger always ran: the redirect rendered before the goroutine wrote
LastStatus=running, so the poll never armed and the page sat static during the
very run just started. Red-proofed twice.
Also closes the golden/controller infra-image drift at the source: infra.Images()
derives from the existing pins and --print-infra-images exposes it, so the golden
bake can stop carrying its own copy. That copy had already drifted — felhom-samba
was never added, so the golden baked 3 of 4, which is why enabling Megosztás
pulled at runtime in the first place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Overwrites REPORT.md per convention. Records the two changes, the four design
decisions behind the accordion (real <button>, landing-pages-survive check,
server-side open state, grid-rows instead of max-height), and — importantly —
what could NOT be verified and why: the demo controller's password is
customer-owned since the claim flow, so the build-server credentials are stale
and a curl-login returns the Bejelentkezés page. The four red-proofed render
tests stand in for the server-side half; the visual leg needs Viktor's browser.
Also notes the fleet gap: v0.146.0 is live on the demo box but the golden still
bakes 0.143.0, deferred to the next session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The collapsible sidebar is a progressive enhancement: the server marks the group
owning the active page with .is-open, so the correct group is open before any JS
runs and stays open if JS never runs. That server-side half is what these tests
pin — the part a browser screenshot confirms only for whichever page happened to
be open when the screenshot was taken.
Four properties, rendered through the real shared layout rather than a hand-built
fragment:
- every sub-page opens its own group, with aria-expanded=true and an .active
toggle, and EXACTLY ONE group open (the count is asserted, not just the group
we expected);
- a page outside any group (dashboard) opens nothing;
- each group's landing page still exists as a sub-link — the property that made
converting the headers from <a> to <button> safe. If someone drops one of
those sub-links the destination becomes unreachable from the sidebar
SILENTLY, because the header still looks clickable;
- the toggle is a real <button> and its aria-controls targets an element that
actually exists.
Red-proofed: removing `{{if $storageOpen}} is-open{{end}}` from layout.html
fails both the open-group assertion and the exactly-one-open count on the two
storage pages, then passes again when restored.
Note on verification: the live authenticated render could NOT be checked from
here — the demo controller's password is customer-owned since the claim flow
(Viktor set it during the 2026-07-18 rehearsal), so the credentials on the build
server are stale and a curl-login returns the Bejelentkezés page. These render
tests exercise the same template through the same loadTemplates() path the server
uses; the visual leg needs Viktor's browser.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
UI-only; no behavioural or backup/restore surface touched. Green: build, vet,
tests all pass; template_id_gate, emoji_gate, native_confirm_gate,
offbox_rename_gate, mojibake_gate, app_row_dedup_gate all PASS.
Scrollbars (style.css): the platform default reads as a bright chunky stripe
against the navy and competes with the content it scrolls. Now thin and
hairline-coloured — scrollbar-width/scrollbar-color for Firefox AND
::-webkit-scrollbar (8px, thumb --line, hover --text-3, --radius) for
WebKit/Blink, because neither alone covers the browsers customers use. The two
surfaces that actually scroll take their own panel background as the track
(.sidebar -> --bg-2, html -> --bg-0) so the gutter never shows as a lighter
channel. Tokens only, no raw hexes.
Collapsible nav groups (layout.html + style.css, vanilla JS, no framework):
Tárhely, Biztonsági mentés and Megosztás become accordions with a chevron;
exactly one open at a time, clicking the open one closes it. Groups without
sub-items are untouched plain links. Hungarian labels unchanged.
- The header is a real <button>, so keyboard and AT reachability come for free
instead of being simulated with tabindex/role on a div. aria-expanded +
aria-controls + a :focus-visible outline.
- Nothing became unreachable when the header stopped being a link: every
group's landing page is ALSO its first sub-item (/storage -> Meghajtók,
/backups -> Áttekintés, /sharing -> Hálózati megosztás). Checked before the
conversion, not assumed.
- Progressive enhancement: the group containing the active page is rendered
open SERVER-SIDE (.is-open), so the right group is open before any JS runs
and stays open if JS never does. The listener only handles clicks.
- No layout jump: collapse animates grid-template-rows 0fr -> 1fr (with
min-height:0 + overflow:hidden on the sub-list) rather than max-height, so
it animates to the content's REAL height and there is no magic number to
drift when a group gains or loses an item — the specific way a max-height
accordion rots. The toggle reserves its 3px active border as transparent so
becoming active adds no width shift. .18s transitions, and both the collapse
and the chevron rotation are disabled under prefers-reduced-motion.
Pre-existing and deliberately NOT bundled: docker_run_volume_path_gate.py still
fails on internal/appexport/estimate.go:179. That is ROADMAP R-29, unrelated to
this change, verified to fail identically on the untouched tree, and R-29 itself
says not to bundle it into an unrelated feature commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The gate landed v0.125.0 (2026-07-13), realVolumeSize landed v0.129.0 (2026-07-14)
-> red for 16 releases unnoticed. Records that the flagged mount is a NAMED VOLUME
(the safe shape) so the fix is an allowlist entry, not a rewrite of correct code.
The prior revision claimed the demo box had no offsite target and that both legs
were unexercised. All three clauses were false on a box that had run a successful
offsite backup minutes earlier.
Evidence: offsite snapshots e0b9d723 (Viktor 12:18:16Z) and 4e2b15ec, tagged
felhom-offbox,_shares, each carrying the payload dir (manifest + passdb.tar, 0600)
plus both share folders. Restore round-trip through the real routes returned a
deleted probe byte-identical and a deleted DEFINITION with its original flags,
without overwriting the two pre-existing files. R-7b is fully PROVEN-LIVE.
Root cause in new REPORT section 7b: the check read a GUESSED settings key
('offbox_target') where the real tag is 'offbox' (settings.go:96), got None for
every field, and read that as 'not configured' instead of 'wrong key'.
Findings, all six red-proof outcomes, deployment, live-validation evidence, and an
explicit list of what was NOT live-exercised (offsite leg + restore round-trip: the
demo box has no offsite target).
Found by LIVE VALIDATION, not by a unit test: the first demo tier-2 run pushed
'Masodlagos mentes elkeszult: _shares' — the reserved key reached Hungarian
customer/operator copy.
Mapped at the SOURCE of the notification (RunSharesTier2's tier2Notify calls) so no
future notifier wiring can reintroduce it, plus DisplayStackName at the main.go
wiring as idempotent defense in depth. Regression test added with a red-proof.
The samba_classify.go SCOPE NOTE and the README KNOWN GAP both described a gap that
R-7b closes; both now describe the sibling-shares-source execution instead.
Part 4 — restore: RestoreSharesScratch + PlaceSharesRestore as SIBLINGS of the
per-app scratch/place pair. Files merged missing-only (never overwriting), each
destination PREFIX-ASSERTED against registered LIVE storage roots; definitions
merged with existing-wins; ReconcileSamba via a seam (backup must not import
stacks); credential restored best-effort into the samba named volume.
New routes POST /backup/shares/{restore,place} + a restore-page entry that renders
'Megosztasok', never the raw reserved key.
Also adds scratchJoin: reconstructing an absolute captured path under a scratch
must strip the volume name rather than rely on filepath.Join.
Part 5 — liveness: EffectiveProtected gains a settings-backed dynamic extra so the
samba CONTAINER (not the stack name — they differ) is watched exactly while sharing
is on. FINDING: the issue -> health 'fail' -> existing health_critical event ->
alert -> Hungarian degradation e-mail path needs NO further change, and introduces
no new event type, so the allowlist gotcha does not apply.
Part 6 — UI: per-tier backup status lines on the Megosztas page (amber only on
deviation). Verified the two warning-prose sites (offbox_capture/tier2_capture)
only ever receive per-app stack names, so no mapping is needed there.
RED-PROOFS RUN AND REVERTED (both fired):
4. prefix-assert removed -> place-guard traversal test FAILS
5. dynamic samba extra removed -> Scenario E enabled-case FAILS
ONE additional restic call tagged [felhom-offbox, _shares] carrying the payload
staging dir + every mandatory share folder. Hooked into runOffboxInternal AFTER the
per-app loop and BEFORE retention, so forget --group-by host,tags covers the _shares
group with no flag change. Reuses resticStep, the caller's repo-ensure and
single-flight, and the SAME enlargement-gate arithmetic.
- quota gate degrades the push to MANIFEST-ONLY, never to nothing
- EnlargedBlocked keeps the RAW _shares key (templates index by it); the display
mapping applies only at the notification + Hungarian-prose boundaries
- OffboxTarget gains SharesLastRun/Status/Count for per-tier page truth
- zero-toggle notice suppressed when the shares leg provided coverage
- reserved-name defense: an app keyed _shares is excluded from the run loudly
RED-PROOFS RUN AND REVERTED (all fired):
1. shares leg appends into the app's argv -> isolation test FAILS
2. mandatory->offsite mapping inverted -> Scenario A + B FAIL
3. manifest-only degradation dropped -> Scenario C FAILS
Sibling shares source for the local cross-drive tier. Reuses the tier2Mirror seam,
selectTier2TargetFrom (narrow source-drive seam extracted from selectTier2Target),
tier2ReconcileRoots (pure extraction), tier2SafeRemove, the marker-LAST discipline
and the recordTier2* helpers. Per-app paths are untouched.
- shares_payload.go: deterministic _shares-manifest.json + best-effort passdb capture
- tier2_shares.go: per-source-drive legs -> cross-drive target, payload, marker LAST
- infra.SambaContainerName/SambaPassdbVolume/Mount: single source of truth for the
container identity (renderer, stacks execs, backup execs, monitor all read it)
- RESERVED-NAME finding: ValidateSMBShareName did NOT exclude a leading underscore,
so "_shares" was an accepted share name. Now refused; RunAllTier2 additionally
skips a "_shares" stack loudly as defense in depth.
- fix: shareSourceDrive returned a slash-normalised path, which made the target
selector's source-drive equality check miss (a group could target its own drive)
Viktor confirmed on the demo box: Network -> FELHOM -> both shares open; an
INTERACTIVE Explorer save into dokumentumok succeeded and landed owned 1000:1000
(force user holds for a real Explorer write, not just a scripted one); a write
into the read-only filmek was refused by Windows with the folder left empty on
disk. Capability-map row flipped to PROVEN-LIVE in felhom.eu.
Records the R-7 slice 1 ship, the four red-proof outcomes, live-validation
evidence, the two bugs live validation caught, and the Part-4 Step-1 enumeration
finding + the reported design fork (share data classified but not yet in a live
backup run — needs a Viktor ruling, suggested R-7b).
Live validation caught it: the 'new folder' flow passed the storage root through
sharingResolvePath, which (correctly) refuses the drive root as a share target —
so share creation silently failed. sharingResolveStorageRoot accepts EXACTLY a
registered live root (strictly tighter) and is used only as the new-folder parent.
Regression test asserts both halves.
The /api/ subtree is routed on the main mux, so the browse case in the web
ServeHTTP switch was shadowed by the apiRouter catch-all and 401'd. Moved to
ServeSharingAPI behind RequireAuth+CsrfProtect, matching /api/storage/.
Found by live validation.
New top-nav category with the Halozati megosztas page: enable/server-name card,
household password, shares table (Nev/Mappa/Irasvedett/Felhomentes/Torles), and
a create flow (new folder under <storage>/shares or an existing folder via the
browse modal). Every customer path goes through sharingResolvePath: absolute ->
EvalSymlinks -> containment in a registered live storage root -> deny-listed
system subtree check -> is-a-directory. Refusals are UNIFORM so the picker is
never a filesystem oracle. Deny-list derived from ProtectedHDDPaths (provably a
subset); the drive root is an exact-match denial so user-data folders under it
stay shareable. samba infra metadata + i-share icon. Gates green.
ClassifiedBinds("samba") resolves from the shares registry instead of catalog
metadata (samba has no .felhom.yml; its binds are absolute share paths). [R4]
Offsite ON -> mandatory (offsite + tier-2); OFF -> optional (tier-2 only);
smb.conf/passdb never classified. Verified through the REAL ComputeCaptureSet
tier filter incl. the negative (optional NOT in offsite). Zero engine edits.
Part-4 Step-1 finding: tier-2 (RunTier2) short-circuits on os.Stat(unitDir)
BEFORE GetStackClassifiedBinds, and the offsite runner enumerates
settings.GetOffboxApps() — both are recovery-unit shaped, which a share-only
infra stack has not. Wiring share data into a live run is therefore more than an
enumeration tweak; reported as a design fork per the STOP clause, not improvised.
ensureSamba joins EnsureBaseStack after filebrowser, gated on SMB.Enabled
(cloudflared conditional precedent); reconcile is idempotent (unchanged config +
running container = ZERO compose calls, asserted via seam). Atomic tmp+fsync+
rename config writes. Password applied via smbpasswd on STDIN (never argv/log/
settings). Disable = compose down, volumes + folders KEPT. samba added to
IsProtectedStack in code (controller.yaml is golden-generated and predates it),
which also makes the app-backup loops correctly skip it.
SMBSettings + SMBShare registry in settings (password never stored — only UserSet);
NetBIOS-safe name validation. Pure infra renderers: RenderSambaConfig (hardened
global block: SMB2 floor, bind interfaces only=lo eth0, disable netbios=no, force
user block) + RenderSambaCompose (network_mode host, pinned image, :ro bind for
read-only shares, passdb volume). Exact smb.conf golden + CRUD/validation tests.
Own pinned alpine image (3.21@sha256:48b0309c) + smbd/nmbd/wsdd/tini. Dumb by
design: smb.conf bind-mounted read-only, no baked name/password, passdb on a
volume. Three-daemon stack per the R-6 spike verdict (nmbd required alongside
wsdd, else Explorer double-click 0x80070035). build-samba-image.sh helper.
The customer sees the guest's current memory + allowed range on the Rendszer settings
page and resizes it. The controller proxies + maps the agent's machine code to Hungarian;
the agent (felhom-agent v0.90.0) enforces every bound and applies live (no reboot).
agentapi: GuestMemory + ResizeMemory; ruled 412 -> *MemoryRefusedError (code+bounds);
pre-0.90 agent 404 -> typed *StatusError. Capability: FeatureGuestMemoryResize +
featureMinAgent 0.90.0 + a featureProbes row (type-asserts GuestMemory so the shared
SupportProber/netAgent are untouched).
UI (internal/web/system_memory_handlers.go, settings_system.html): "Szerver memoria (RAM)"
card + number input; POST /api/system/memory/resize -> capability gate -> agent -> flash.
JS confirm only on shrink. Code->Hungarian map; agent-outdated hides the control;
agent-unreachable falls back to the guest's /proc/meminfo. Agent English never shown raw.
Tests: agentapi (decode, 404, refusal-code, capability table) + web handler (success/
below_usage_floor/agent_outdated). Gates + go build/vet/test all pass.
- F6: POST /api/storage/init runs format→mount→register as a DETACHED single-flight job (context.Background, netAddState shape) the wizard polls via GET /api/storage/init/status; 3-step progress; register-last marker-last crash-safety. Fixes the client-disconnect-aborts-mount bug. No agent change (chain reaches FileBrowser sync = controller-only). Red-proof TestStorageInit_DetachedSurvivesClientDisconnect.
- F7: storage_init/attach Vissza → /storage (was /settings). Test TestStorageWizardBackAnchors_PointToStorage.
report.Waiter holds a hanging GET against hub /api/v1/wait?gen=N (same hub
URL+key as the pusher). On a generation change it fires the v0.139.0
report.Trigger — nothing else; the report ACK delivers everything through the
unchanged machinery. No overall client timeout (held GET); per-request ctx
bounds a dead connection. First-observation records-not-fires; same-gen
timeout fires nothing; errors (incl. 404 from a pre-v0.58.0 hub) back off
5s->5min while the 15-min cycle reconciles. Wired beside the trigger under the
same hubPusher!=nil && Hub.Enabled gate.
Red-proof: disable the baseline branch -> first observation fires
(TestWaiter_FirstObservationRecordsNoFire), run-fail-reverted.
Copy soften: backups_remote/escrow "néhány másodperc, legfeljebb 15 perc".
Pairs with hub v0.58.0. Grounding:
felhom.eu/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md
Out-of-cycle pushes at 17:42:19/17:42:34Z (quiet-window + min-interval
pacing exactly as designed, trailing edge carried the final state);
scheduled 15-min cycle unaffected (17:49:00Z tick on both ends).
Red-proof outcomes and the NOT-yet-live-validated list recorded.
After a completed escrow ceremony the Távoli mentés page showed the yellow
"Helyreállítási kód szükséges" card for ~15 min until the next hub-report ACK
flipped pending→escrowed. Phase-0 diagnosis (read-only) = verdict A (report-cycle
lag), already resolved on the demo box (escrow_state:"escrowed"); hub Hypothesis B
verified false (SaveHostEscrow ON CONFLICT already clears stale_at on upload) → no
hub change.
- settings.OffboxTarget.CeremonyCompletedAt: stamped on the recovery-code claim,
zeroed on the auto-confirmer Flip + the deprecated manual confirm; persisted.
- web/handlers.go: offboxCeremonyWaitState + escrowCeremonyGraceWindow (35m).
- backups_remote.html: info "megerősítésre vár, legfeljebb 15 perc" card → warn
"a megerősítés nem érkezett meg" past the window. Existing branches untouched.
- backups_escrow.html: "Mi történik ezután?" note on the wizard's final step.
- Test web/escrow_wait_state_test.go (truth table + red-proof recorded in REPORT).
No scheduler/agent/hub/endpoint changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
Saving the notifications form with a blank email box while events are enabled
wiped the customer's hub-side alert address (SyncPreferences pushed empty) —
the 2026-07-15 demo incident. settingsNotificationsHandler now refuses that
save before SetNotificationPrefs + hub sync, re-renders a Hungarian error, and
repaints the submitted checkboxes. Empty email + zero events (clear-all) still
proceeds. Tests + red-proof (remove guard -> stored email wiped to empty).
SQ6 over-capture FIXED for classified apps: the userdata root tar is exclude-scoped (keeps only
ancestors/descendants of a SELECTED bind relpath — R1-C, the tier2Reconcile keep-rule); no selected
userdata bind → no root tar (radarr state-only). New appbackup.ComputeFabBuckets (shared
resolveGuardCollapse pipeline; class buckets; guards over ALL classes; no cross-bucket containment).
appexport/fabplan.go: computeFabPlan + tarDirectoryExcluding + fabEstimateSplit. ExportRequest gains
DeselectOptional/OptInExcluded (both start handlers — two-call-site); mandatory is a server-side
floor. Manifest v1 + import UNTOUCHED; legacy apps byte-identical to v0.130.0. Estimate additive
class split; export UI: locked-mandatory/optional-checkboxes/excluded-opt-in + two-number warning.
All 6 §10 red-proofs verified. 6D Accept #1 now runs against this shape.
tier2_capture.go: classified apps get TierSecondary per-bind legs (paperless copy shrinks — export
drops); legacy apps keep the byte-identical resolver set. v2 relpath-mirroring layout
(backups/secondary/<stack>/{marker LAST, recovery-unit/, hdd/<rel>/, userdata/<rel>/}); N>1 native
(errTier2MultiDir/tier2AppDataName deleted). Migration=delete-and-rebuild + reconcile; all RemoveAll
via tier2SafeRemove (refuses outside backups/secondary/). SSD=state-only tier. selectTier2Target
never picks network storage (pinned+auto, F-6C-1). Restore reads v2 behind a marker gate.
Part 0: offbox_enlarge_blocked is a persisted one-time Load seed (opt-out sticks), not a getter
append. Part 0.5: offsite restore scratch prefers a local (non-network) path.
Full v2 test suite + all 10 §10 red-proofs verified. Destructive writes bounded to backups/secondary/.
PlaceOffsiteRestore: live target via raw GetStackHDDPath not AppNamespaceRoot (F-3a-1a: no SSD
merge; undeployed refused), placement headroom gate (F-3a-1b), stat pre-pass over all placements
before any copy (F-3a-4: no partial writes), scratch removed on success/kept on failure (F-3a-2).
mapOffsiteRestorePaths refuses the namespace root itself (F-3a-3).
Delivery chain: DefaultEnabledEvents + GetNotificationPrefs append-if-absent migration + settings
checkbox + handler slice; paired with hub v0.55.0 allowlist (no customerMessages entry — raw
dynamic message survives). +8 tests; all 6 controller §10 red-proofs verified.
Task 2 of the backup-classification-redesign arc. Ships the referential-
coupling classification as DATA + PARSER + PURE CLASSIFIER, deliberately
inert — no backup tier changes behavior. Task 3 (tier policy engine) and
Task 4 (manual .fab UI) consume it.
- appbackup/classify.go: BackupSpec/BindSpec/ComposeBind/ClassifiedBind;
ClassifyBinds (SQ5 two-level default — explicit beats :ro; unlisted
writable→mandatory, unlisted :ro→excluded; nil spec→legacy/false);
ValidateBackupSpec (whole-block-reject on any defect, first defect named).
- stacks/classify_binds.go: ParseComposeClassifiableBinds — ${VAR}-relative
binds + :ro flag (NOT ParseComposeHDDMounts/ExportDataMounts, the traps).
- Metadata.Backup + LoadMetadata as the single validation choke point (bad
catalog block → nil + one ERROR → legacy, within one sync cycle).
- Manager.ClassifiedBinds + StackDataProvider.GetStackClassifiedBinds seam
(delegated by stackAdapter, nil-stubbed in every fake) — wired + tested
now so Task 3 consumes a tested seam.
INERT: full pre-existing suite green with zero test-logic edits. +14 tests;
red-proofs RP-1..RP-4 confirmed. The 13 catalog backup: blocks ship in the
same app-catalog change (this controller deploys first).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
Both guests live+healthy on 0.131.0. Live functional legs (paperless
deploy → tier-2 backup → marker restore → storage page → F-S3 migration)
deferred to Viktor's supervised session — paperless-ngx is undeployed on
the demo and the API is container-internal; documented with rationale.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
The controller assumed an app's HDD appdata dir is always appdata/<stackName>.
paperless-ngx writes appdata/paperless (stack paperless-ngx), so every consumer
keying by stack name silently missed it via a stat-and-skip. One canonical
resolver appbackup.AppDataDirNames derives the real dir name(s) from the app's
compose ${HDD_PATH} binds; all consumers use it.
- F-S2 (tier-2): RunTier2 mirrors the resolved appdata/<name> (paperless docs
got NO tier-2 copy before). Tier2Info size + RestoreTier2Files live dir use it.
WARN when a declared appdata dir is absent. New tier2Mirror seam.
- F-S3 (migrate, NEW): all six per-app appdata legs (collision/size/copy/verify/
cleanup/skip-set) now loop resolved names. scope="app" migration of paperless
previously copied nothing and left an empty media dir (scope="all" was saved by
the merge walk). WARN on missing declared dir in the copy leg.
- Multi-dir (N>1) refusal: tier-2 backup/info/restore refuse loudly (Hungarian);
migrate supports N. No catalog app hits it today; lifted by Task 3.
- Display: storage page sums resolved dirs.
- Truth repair: the v0.130.0 "tier-2 copies the namespace wholesale" claim is
false; corrected in CHANGELOG + main.go export-adapter comment.
+9 tests; red-proofs RP-1..RP-5 all confirmed. Controller-only, no agent/hub
coupling. Task 1 of the backup-classification-redesign arc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
handleNetStorageRemove now refuses (409, Hungarian, names the apps) when any
DEPLOYED stack's HDD_PATH is the share root or a subpath of it — the C6B live
event removed campaign6 under a running sonarr, and the agent's tolerated
best-effort stop steps then deleted the unit files under the busy mount,
leaving an unreapable orphaned autofs mount until host reboot. The guard cuts
that chain off at the product flow. The remove handler resolves the agent via
the netAgent seam (netAgentForAdd), making the negative control testable.
NOTE: the agent-side residual (tolerate-and-continue stop in felhom-agent
netmount.go RemoveNetworkMount) is out of this controller-only task's scope —
flagged in REPORT for a follow-up agent task. Red-proof recorded: disabling
the guard returns the live pre-fix removed:true.
assertBundleDataComplete's claimed-tar checks pass trivially when discovery
claims nothing (0 claims -> 0 checks) — the exact blind spot that shipped a
4.17 GB app as a 2308-byte config-only bundle. A needs_hdd manifest with
neither HDD data nor volume data now fails the job with an honest Hungarian
error. Red-proof recorded: removing the assertion returns scenario D to
silent hollow success.
executeExport no longer either/or gates user data on needs_hdd — a needs_hdd
app bundles BOTH its HDD mounts and its named volumes (sonarr_config = the
whole app DB was silently dropped pre-fix). exportHDDData returns error and
fails LOUDLY on a basename collision between mounts (the manifest keys tars by
basename; the old code silently overwrote the first tar). EstimateExport made
additive to match, so the fits-on-dest gate counts both. Round-trip placement
test proves a userdata tar restores to <HDD_PATH>/userdata through the
untouched import mapping. Red-proofs recorded: either/or revert fails
scenario A; collision-check removal fails the collision test.
exportAdapter.GetStackHDDMounts now returns stacks.ExportDataMounts — the
${HDD_PATH} binds UNIONed with the ${USERDATA_PATH} ROOT (captured at the
root, not per-bind, so the manifest's basename keying round-trips through the
existing import mapping without touching restore). Containment-aware dedupe
both directions. The backup-side stackAdapter is intentionally unchanged.
Red-proof: pre-fix behavior fails TestExportDataMounts_UserdataConvention/
MixedBindsUnion/LiteralUserdataBindDeduped (run->fail->revert recorded).
Only Observe-sourced drives ever carried class, so legacy PVE dir:-backed USB
drives showed 'lassú' while registry-sourced ones never did. The card already
carries the USB type tag. Hub-report ClassHint UNCHANGED (UI-only). Pinned by
TestStorageTemplate_USBClassBadgeSuppressed (red-proof run).
- offboxWarningDisplay(lastWarning, toggledCount) — pure DISPLAY pick, no state mutation:
a persisted 'nincs mentésre jelölt alkalmazás' run-result is replaced by
'A kijelölés módosult az utolsó futás óta — a következő távoli mentés már tartalmazza.'
once ≥1 app is toggled; 0 toggled keeps the v0.123.0 honesty verbatim; every other
warning (quota, partial) passes through untouched
- replacement renders NEUTRAL (exception-color principle: reassurance, not deviation);
the untouched original keeps the warn color
- unit + render tests; red-proven (pick removed → 1-enabled case fails at both levels)
- scans all templates + Go sources (291 files) for the double-encoding signature chars
(Ã Â Ă ă ˘ ˇ) + strict-UTF-8 decode; allowlist ZERO by design; Python per the
multibyte rule (this task exists partly because of a bash-era mojibake sweep)
- byte-level sweep result: the committed source contains NO mojibake — the 'Tárhely'-class
text on the live import page is the felhom-usb DRIVE-LABEL DATA (settings.json on the
demo box; default-label pattern settings.go GenerateLabel), repaired via the label-edit
UI in the live-validation step, not by code
- red-proven: reintroduced 'Tárhely' into app_import.html → exit 1 naming file:line; reverted → 0
- backups_restore: placeholder 'Opcionális jelszó' + helper under the field
('Üresen hagyva a csomag titkosítás nélkül készül.') — the long clipped placeholder is gone
- app_import: the bundle-password input picks up the standard .form-input class
(was a bare browser-default input)
- inframeta.go: static display-only map (name/description/Linked) + infraMeta template func;
filebrowser is the ONLY Linked stack (files.<domain> Megnyitás on the dashboard row;
the stacks card already links via Subdomains)
- generic infra icon: embedded /static/infra-logo.svg (Lucide-style server, monochrome)
wired through the app-row FallbackIcon slot + the stacks-card onerror chain
- dashboard rows + app cards: infra name + description + existing Védett chip
- guarded WRONG outcome: no customer link for cloudflared/traefik — render tests count
exactly ONE https:// link; red-proven (Linked:true on cloudflared → test FAILS)
The customer sets + owns the dashboard password via a hub-emailed one-time
claim code. An unclaimed box (code hash present, no password) serves ONLY the
claim page — every other route → claim page (302) or 401, so a Day-0 box is
never open on the internet. A set password disables the gate (auth wins).
Reset rides the same code engine (login "Elfelejtett jelszó"). Legacy-open
(no password, no hash) shows a red transition banner until the hub delivers a
hash. Report ACK caches the code state idempotently by generation; report
carries claimed (set-only). --print-reset-code root escape hatch. Requires
hub v0.50.0. Gate-coverage signature test + 4 red-proofs proven.
F8: share row fuses the shipped v0.117 classifier (fuseNetHealth) so it agrees with
the stacks stub badge (one classification, two surfaces). F4: mapped_uid range check
1..65533 at the door. §3 fork = option B (controller-only). Live-validated on 9201.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
IsRunning flips before the restore goroutine parks in StopStack; under parallel
load the ==1 assertion raced it and read 0. Wait for the first stop to land, then
assert — the red-proof property (no-fast-path mutant → count 2) is preserved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
- networkStoragePageData threads NetAddSupport (yes/no/unknown) via the short-budget
netAddSupport probe (2 s ctx + cache — a down agent cannot stall the page)
- storage_network.html: support=no replaces the add form with the Hungarian
alert-warning banner; share list + remove render in every state; yes/unknown
render the form unchanged; load-time JS guarded for the formless render
- T5 render test (banner/no-form on 'no', form on yes/unknown); red-proof RP5 run
(conditional dropped → banner assertions fail) and reverted
- template_id_gate + emoji_gate green
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Kills the details/summary-button hack and the nonexistent form-row/form-input
classes (the unstyled-look root cause); rebuilds on the storage_attach pattern
(settings-card / form-group / form-control / form-actions / alert-*). SMB listed
first (consumer reality), NFS two-recipe guidance (map-all-users simple recipe +
full-fidelity anonuid=<uid+100000> with a live computed host-id), §3.2 staged
poll progress (Kapcsolódás/Csatolási teszt/Írásteszt/Regisztrálás), categorized
Hungarian errors with a collapsible raw-detail block, orphan rows with an Árva
badge + remove-only action. C8 render smoke guards the class regression.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Root cause (live on Peti's box): the updater piggybacked on Git Sync creds and refused when absent,
but the registry serves the public package anonymously (verified 2026-07-10). Credentials become what
they were meant to be — optional, for private catalogs only.
- queryRegistry: both creds empty → anonymous flow (plain GET → parse WWW-Authenticate realm/service
from the header, never hardcoded → credential-free token → Bearer retry); creds present → BasicAuth
path unchanged; half-configured pair → loud incomplete-credentials error
- pullImage: no creds → skip docker login entirely (docker's native anonymous flow); denied anonymous
access → clear 'registry denied anonymous access — a private registry requires Git Sync credentials'
- settings page: 'Registry: nyilvános (hitelesítés nélkül) / hitelesített' mode line — credential-less
is no longer an error state; DryRun.PullCapable counts anonymous as capable
- tests: fake registry httptest token dance (zero creds, no auth on token request, correct scope),
creds path unchanged (BasicAuth, no dance), both denial paths, WWW-Authenticate parser table
(quoted/bare/order/comma-in-quotes/missing-realm), fake-runner pull tests (no login invoked
anonymously; login/pull/logout order + stdin token with creds; partial creds refuse)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
resticStep escalates a restic lock error to `unlock --remove-all` + one
retry (safe: single-writer repo — sub-account isolation + single-flight
mutex); plain `unlock` is stale-only and can't clear a crash lock across a
container-hostname change. Pre-run stale unlock hygiene on run+restore.
C1: NewManager flips a persisted LastStatus=running to a truthful error.
Both red-proofed (A reproduces the exact campaign backup failure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The QuotaGB hash change triggered a live re-apply that demoted the
escrowed demo to pending and wiped its runtime status. ApplyOffsiteTarget
now carries over EscrowState (custody tracks the preserved repo password,
not the coords) + status fields; fresh guests still land pending.
Red-proofed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
A hub-side quota raise now reaches the target: the marker hash changes and
the bridge re-applies via key-auth-first (no password consumed). Test:
quota-only change remaps the new quota with a panicking consumer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
QuotaGB rides the descriptor into OffboxTarget; RepoSizeBytes persisted
from restic stats. Pre-run gate: >=100% refuses NEW backups (Hungarian
notice + operator alert) but prune STILL runs (red-proofed) and restore is
never gated; >=80% warns. /backups usage bar (quota>0 only). The hub
report gains the non-secret offsite status object for the OffsiteChecker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
EscrowAutoConfirmer flips pending->escrowed ONLY when sha256(local repo
password) matches the ACK's restic_pw_sha256 (blob-presence alone never
confirms — red-proofed). Mismatch warns once per hash naming the ceremony;
never un-confirms; wipes the staged secret on flip. Pinned cross-repo hash
vector; manual confirm deprecated to a legacy-blob fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Key-auth-first: a KeyAuthProber seam lets the bridge skip consume+install
when the already-installed key still authenticates (pinned to the freshly
verified host key) — descriptor changes on provisioned guests no longer
loop on consume-404. Fingerprint verify still precedes everything.
Wipe-on-escrowed: confirm-escrow now calls the agent's new
DELETE /escrow/stage-secret (v0.78.0) best-effort, closing the hygiene gap
where a ceremony-less confirm left the staged password file behind.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live finding: SFTP-mode ssh-copy-id mktemp's under ~/.ssh; the container
image has no /root/.ssh, so the install died locally AFTER the one-time
password was consumed (fail-safe held; password never transmitted).
Probe confirmed the pinned known_hosts + StrictHostKeyChecking=yes chain
is sound against the real box once ~/.ssh exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The SSHCopyIDInstaller used StrictHostKeyChecking=accept-new on the ssh-copy-id
and sftp-verify connections, so even though the bridge verifies the box host-key
fingerprint against the hub descriptor BEFORE installing, the actual install
connection was not pinned to that verified key — a MITM could substitute a
different key in the gap between the scan and the install (TOCTOU).
Now the bridge threads the scanner-verified known_hosts line into KeyInstaller,
which writes it to a temp known_hosts and connects with StrictHostKeyChecking=yes
+ UserKnownHostsFile — the install/verify sessions refuse any key but the one the
bridge already matched. Empty known_hosts now refuses to install.
Test asserts the installer receives the pinned known_hosts; red-proofed by passing
an empty line (the pre-fix TOFU shape) → test fails. Addresses the security-review
"host-key TOFU after verify" finding on internal/offsiteapply/seams.go.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
On startup reconcile the hub-served offsite: descriptor into a key-only offbox
target. internal/offsiteapply.Bridge: verify-pin box host key vs host_fingerprint
(NO blind TOFU) → consume the one-time password (single-use, never logged) →
sshpass ssh-copy-id -s -f install + verify → configure offbox → EscrowState=pending
(fork-4 via Manager.ApplyOffsiteTarget) → persist a descriptor-hash marker LAST.
Idempotent + fail-safe. Seams faked in tests; both red-proofs run+reverted.
Dockerfile + sshpass. NOT yet live-applied (supervised end-to-end next runbook).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pairs with agent v0.77.0. StageEscrowSecret pushes the repo password to the
agent (POST /escrow/stage-secret) at offsite-enable → EscrowState="pending".
Atomicity gate: RunOffboxBackup (scheduler + handler) refuses until
EscrowState="escrowed" (operator POST /backup/offbox/confirm-escrow after the
escrow ceremony) — no un-recoverable offsite ciphertext can exist. DR:
POST /backup/offbox/inject-password pre-places a recovered 64-hex password 0600
(honored by WriteOffboxSecrets' IsNotExist guard; refuses clobber without
force). DR recipe gains non-secret offsite_restic coords (DRResticCoord); SFTP
key regenerated at DR, not escrowed. New settings.OffboxTarget.EscrowState.
Tests + atomicity & inject companion red-proofs green; UI gates pass. NOT yet
live-validated (supervised ceremony).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
offbox located each toggled app's recovery unit via AppNamespaceRoot→GetAppDrivePath,
which reads the app's LIVE app.yaml HDD_PATH and silently falls back to systemDataPath
when the app isn't deployed → looked on the wrong drive, backed up nothing, reported
ok/0 (DIAG root cause). Now:
- discoverOffboxUnit/offboxCandidateNSRoots scan the durable storage registry
(schedulable non-decommissioned paths ∪ systemDataPath) for backups/primary/<app>,
independent of deploy state; newest-by-manifest-CreatedAt wins on drive churn.
- RunOffboxBackup: runOffboxInternal returns (backedUp, missing, err); 0-of-N toggled →
hard error + operator alert; partial → ok + new OffboxTarget.LastWarning (shown on
/backups, preserved across config edit).
- AppNamespaceRoot + primary WRITE paths unchanged.
- 6 non-hollow tests (A-E + edge) + both companion red-proofs run (reverted).
- NOT yet live-validated against the Storage Box (spike creds torn down).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
loadAndParse + LoadFromBytes ran os.ExpandEnv over the whole YAML before parse;
a bcrypt hash ($2a$10$...) is full of $word sequences that get replaced with
empty env values, silently corrupting web.password_hash on load (a silent
auth-integrity bug: $2a$10$N9qo8uL... -> "a0"). Remove both ExpandEnv calls;
parse raw bytes. The typed applyEnvOverrides path (FELHOM_WEB_PASSWORD_HASH) is
the sanctioned env mechanism and is unchanged. Tests + red-proof.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Re-adjudicates F4: /backup/restore, /backup/tier2/restore, /backup/offbox/restore
blocked the HTTP request until completion, so through cloudflared's 100s cap a
customer got an error page while the restore succeeded (offbox worse — bounded
on r.Context(), canceling the SFTP restore mid-flight). Convert all three to the
offboxRun async shape: fast-path IsRunning refuse, background goroutine
(offbox ctx off r.Context() -> Background+30m), instant redirect. Add mutex-
guarded op-status (opstatus.go) + GET /api/backup/restore-status + a 3s-polling
backups.html banner (neutral running, red on failure). Restore single-flight
unchanged. Tests + red-proof (sync handler blocks indefinitely vs <500ms async).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
EjectDisk/Decommission switched from c.post (drops non-2xx body) to
postWithStatus + shared refusalError, so the agent's informative 403 body
("…decommission refused (role: X)") reaches the operator instead of a bare
"HTTP 403" (campaign F2 evidence gap). Generic post + other callers untouched.
Tests T-D1/T-D2/T-D3 + ok:false case; T-D1 red-proof shows the pre-fix bare
"HTTP 403". Bundles the v0.101.0 CHANGELOG entry (this + the F3 sync deadline).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
runGitInDir had no context: a hanging remote parked the sync goroutine in
cmd.Run(), the doSync defer never ran, `syncing` stayed true, and every manual +
periodic sync was refused with "Szinkronizálás már folyamatban" until a
controller restart. Each git command now runs under exec.CommandContext with a
fresh per-command gitCmdTimeout (120s); the deadline error names the timeout and
the (masked) git args. Debounce and failed-sync-arms-debounce unchanged.
Tests: T-C1 cancelled-context kills the subprocess promptly (red-proof: pre-fix
exec.Command shape runs to completion → test FAILS); T-C2 failed sync releases
`syncing` and a post-debounce retry EXECUTES.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
- Endpoint next to /backup/restore; handler mirrors backupRestoreHandler
(ParseForm → validStackName → backupMgr guard → WARN with RemoteAddr →
RestoreTier2Files → flash). Flash strings: "<stack>: N fájl visszaállítva a
másodlagos másolatból." / "Nincs hiányzó fájl — minden fájl megvan a helyén."
/ "Fájl-visszaállítás sikertelen: <err>" (refusals carry the Hungarian
reasons from the engine).
- backups.html: the button on the healthy Tier-2 layer row only (the
Tier2Configured branch already excludes disconnected/inactive; additionally
gated on Tier2LastRun), inline POST form with CSRF + confirm dialog naming
the additive-only semantics and the last-copy timestamp. Template gates
(id + emoji) green.
- Handler guard test (C6): traversal/empty → exact Hungarian flash, no work
started (nil backupMgr would panic if reached).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Closes the engine half of drill finding F2: user files under appdata/<stack>
had no customer recovery path (operator copy-back only).
- RestoreTier2Files(stack): single-flight with backup/restore; ALL refusals
before any stop (no Tier-2 record / LastRun empty / copy dir absent →
"nincs másodlagos fájlmásolat"; Tier-2 drive disconnected; live drive
disconnected/decommissioned — Hungarian, flash-ready); source is the
RECORDED CrossDriveBackup.DestinationPath (never a fresh selectTier2Target);
stop → copy → start → waitForHealthy; copy/restart errors surface (F17).
- rsyncRestoreMissing: rsyncMirror's exec shape with the OPPOSITE-direction
flags: -a --ignore-existing --itemize-changes — existing live files are
never overwritten, nothing is ever deleted (the --delete trap this task
exists to avoid). Count = ">f" itemize lines (pure countRestoredFiles).
- restoreFilesCopier seam so orchestration tests never shell out; the one
FS-level test of the real rsync is LookPath-guarded (runs on the Linux
build server + live validation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The proceed-path for a missing RESETTABLE secret redeployed the app with the
secret blank (compose "Defaulting to a blank string" → exit 1, live-hit in the
2026-07-04 drill Phase 5). Now the restore generates a fresh credential instead:
- stacks.Manager.GenerateSecretForField: replacement value from the field's
catalog generate spec via the deploy flow's generateValue (no logic copied);
refuses data-keys (defense-in-depth), spec-less and non-secret fields.
- backup.Manager.SetSecretGenerator seam (wired in main.go), consulted in
RestoreFromRecoveryUnit AFTER the untouched fail-closed gate, for missing
names NOT in DataKeyEnvVars. The generated value rides fullEnv into
RecreateStackFromUnit → RedeployFromEnv → SaveAppConfig, so it persists
encrypted in the guest app.yaml and round-trips on the next backup/restore
(no second write path). reconcileRestoreSecrets stays pure and untouched.
- WARNs now discriminate: "generated replacement for X (credential was reset)"
vs "X unrecoverable and has no generator — app may fail to start". Values are
never logged (asserted in test).
- Residual case (documented, not pretended away): if a restored volume tar
carries the OLD internal credential hash, the app may still fail auth until a
manual in-DB reset — generation fully fixes only the fresh-init case.
Companion red-proof: pre-fix behaviour (generation skipped) fails
TestRestoreGeneratesMissingResettableSecret on the non-empty DB_PASSWORD
assertion (verified, reverted). Data-key gate proven unreachable by generation
in TestRestoreGenerationNeverReachesDataKeys.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
DumpAppVolumesSafe had NO production caller: no trigger ever produced
volume-dumps/, so named-volume app data (e.g. nextcloud's html volume) was never
captured into the recovery unit and the granular restore silently restored
nothing for class-B data (drill finding F3).
- runVolumeDumps: per-stack loop in runDBDumpsInternal, BEFORE
captureAllRecoveryUnits (so manifests enumerate the fresh tars). Gate order is
load-bearing: protected-stack and volume-check gates precede DumpAppVolumesSafe
(which stops the stack before its own check — unconditional calls would bounce
every volume-less app nightly). Disconnected/decommissioned drives skip with
the same summary style as the DB loop.
- No silent partials: a per-stack failure lands as a FAIL summary entry, flips
Success, and fails the run ("some backup steps failed: ..."), without aborting
the other stacks.
- Zero-DB early return removed: volume-bearing apps without a database still get
their class-B dump + unit refresh.
- dumpVolumesSafe seam (same style as the F17 discoverDBs/importDBDump seams) so
the gating is unit-tested without Docker. Companion red-proof: neutering the
volume gate fails TestRunVolumeDumps_GatesPrecedeDump (dump fired for the
volume-less stack) and _VolumelessNeverStopped (verified, reverted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The backups.html restore panel fetched /api/backup/snapshots (a restic-era route
that no longer existed), so the snapshot dropdown never populated and the
"Visszaállítás indítása" button could never enable — customers could not restore
anything from the UI (drill finding F1, DRILL-appdata-restore-2026-07-04).
- backup.Manager.ListRestorePoints: the keep-side restore has exactly ONE restore
point per app (the current recovery unit); time = newest artifact mtime among
manifest/db-dumps/volume-dumps; tier always 1 (Tier-2 copies are NOT restorable
via POST /backup/restore — never listed); drive_label from the storage registry,
empty for the SSD fallback.
- api: /backup/snapshots route + validStackParam guard (same semantics as
web.validStackName; traversal → 400, unknown stack → 404, no unit → ok+[]).
- Tests dispatch through Router.ServeHTTP (the bug WAS a missing route) + unit
tests for newest-mtime/label/empty semantics. Companion red-proof: hollow
always-[] implementation fails TestListRestorePoints_UnitOnDisk +
TestBackupSnapshots_UnitOnDisk (verified, reverted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Points to the DRILL audit in felhom.eu. Flags F1 (HIGH): the keep-side
restore UI is dead because /api/backup/snapshots has no handler, so the
"Visszaállítás indítása" button never enables (server-side POST /backup/restore
works). Validation-only run; no code changed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
List-removal only deletes the registry entry — it's the undo of a
manual path add. On an enrolled drive (/mnt/felhom-drives/) it leaves a
de-registered-but-still-bound limbo nobody wants; the real lifecycle
there is Biztonságos leválasztás / Végleges leszerelés. New
StoragePathView.IsEnrolled gates the button; manual paths keep it; the
decommissioned-branch removal (final cleanup) is unchanged. Endpoint
untouched.
Two Leválasztás buttons rendered per USB drive (registry safe-disconnect
+ the agent eject the D1 merge carried over like-for-like) and four
near-synonymous labels covered very different operations. The agent
eject now hides when the card offers the registry safe-disconnect (one
detach per card; non-USB + unregistered drives keep eject). Labels are
self-documenting with explanatory title tooltips: Új telepítések
letiltása/engedélyezése, Biztonságos leválasztás, Eltávolítás a
listából, Végleges leszerelés, Formázás…. Endpoints and semantics
unchanged.
The .drive-agent-extra slot stacked the agent tag row and the agent
action row with no vertical gap (and sat flush against the registry
action row). The slot is now a flex column with a .6rem gap + top
margin, hidden when empty; enrichCard's inline margin removed.
User feedback on D1: the NAS-add button and the local-drive enrollment
buttons sat side by side with no separation — two different storage
classes interleaved on one page. Now two subpages under Tárhely:
- /storage — Tárhely — Meghajtók: physical drive registry, migrate,
wizard entry points (Új meghajtó inicializálása / Meglévő meghajtó
csatolása), unified agent view, manual add. The enrollment buttons
now live unambiguously in the local-drive context.
- /storage/network — Tárhely — Hálózati tárhely (NAS): the NAS share
list (NAS-megosztások) + add form + its JS (moved verbatim, incl.
its own openDialog copy for the remove overlay).
- layout.html: Tárhely main-nav item gains two always-visible nested
sub-links (Meghajtók / Hálózati tárhely, .nav-links-nested CSS);
parent stays highlighted on both.
- handlers.go: NetworkStoragePaths moves out of storagePageData into
the new networkStoragePageData (page key storage-network) +
storageNetworkPageHandler; GET /storage/network route.
- Tests: /storage must NOT render the NAS section, /storage/network
renders it and nothing drive-related; inventory + no-native-confirm
scans cover the new template. Both template gates green.
- The D0 grep gate false-negatived multibyte emoji on Windows (its zero
was wrong). scripts/emoji_gate.py scans by Unicode codepoint; it found
8 survivors: backups.html (📁→file-text icon, 🔄 restore-info text),
debug.html (🔄→'fut', 🔒→'titkosított'), deploy.html (📦→upload icon,
★ default-marker → '(alapértelmezett)'), storage.html (📦→upload icon,
📦 in the migrate-target option → plain text). All → sprite icons or
plain words.
- Gates: scripts/emoji_gate.py = 0; new Go TestNoEmojiInTemplates
codepoint scan keeps it enforced (allowlist: ✓✗✔✘•●○■▶ monochrome
text marks). go build/vet/test ./... green (18 pkgs).
- storage.html: the registry cards and the separate 'Meghajtók (ügynök
nézet)' merge into ONE view. Each connected registry card gets an
#agent-extra-<path> slot; the enrichment JS joins the agent /api/disks
list on mount path and decorates the matching card in place (role tag
via i-lock, drive class, durable-id mono line, agent-only
register/eject/wipe actions). Two extra groups render below:
'Rendszermeghajtók' (system/backup — read-only, lock tag, NO actions)
and 'Nem regisztrált meghajtók' (unregistered user-data — register
action only). Agent-down: one warn note into #agent-warn-note, all
registry cards still render from server data (graceful degradation).
The agent-view helpers now emit design-system .tag markup instead of
.badge (roleTag/classTag/dataTag/regTag/appBackingTag); the 🔒 lock
emoji is gone (sprite i-lock).
- Overlay migration: every native confirm()/prompt() on the four pages
routes through a light .confirm-overlay dialog (openDialog; texts
verbatim) — storage remove forms, netStorageRemove, storageMigrateAll,
storageDisconnect, storageDecommission (migrate + the type-to-confirm
anyway branch preserved like-for-like), storageReEnroll; and on the
system page triggerUpdate + controller/server restart; on the security
page the two geo Hungary-removal confirms. Scenario F grep: zero
native confirm/prompt in the four templates.
- Deleted the now-orphaned .badge-lock/.lock-ico CSS (grep-zero first).
- Tests: no-native-confirm scan, agent-down warn-note static assertion;
integrity gate stays green.
- settings.html (1451 lines) deleted; sections moved verbatim into
settings_system.html (Rendszer konfiguráció, Verzió és frissítés,
Vezérlő/Kiszolgáló újraindítása + update/restart JS),
settings_notifications.html (Értesítések, Alkalmazás-email),
settings_security.html (Jelszó módosítás, Földrajzi korlátozás + geo
JS, Vészhelyzeti információk — heading + section copy accents fixed),
storage.html (Adattárolók, NAS, migrate progress, agent view + all
storage JS; wizard entry links now /storage/init|attach with sprite
icons instead of emoji). The NAS + migrate sections were nested inside
{{if .StoragePaths}} in the monolith and vanished with zero drives —
now unconditional on /storage.
- layout.html: Tárhely main-nav item (hard-drive icon) + the
'Beállítások' sidebar group with Rendszer / Értesítések / Biztonság és
hozzáférés sub-links (active-state per page key); orphaned
.sidebar-settings-link CSS deleted (grep-zero), .nav-group-label /
.nav-links-sub added.
- Handlers wired to their own builders + templates; the legacy
settingsData() merge deleted.
- scripts/template_id_gate.py: the §10 JS element-ID integrity gate
(getElementById/querySelector('#…') must resolve in the SAME template;
JS-created + template-parameterized IDs handled; layout modal IDs
allowlisted). Red-proven: a storage function planted in the
notifications template failed the gate with 'static #migrate-progress
not defined'.
- Tests: per-page section markers + cross-leak assertions, h3 section
inventory (all 11 old headings accounted for; typo rename asserted).
- server.go: GET /storage (Tárhely page), GET /settings/notifications
(GET->page, POST->save dispatch on the same path), GET
/settings/security; the enrollment wizards move to /storage/init +
/storage/attach with 301s from the old /settings/storage/* URLs.
- handlers.go: settingsData() decomposed into settingsBaseData +
systemPageData / storagePageData / notificationsPageData /
securityPageData; the legacy merge remains only while the monolithic
settings.html exists (Part 2 deletes it). All five storage action
redirects (add/remove/default/schedulable/label) now land on
/storage?storage_msg=... (incl. the two error-branch redirects).
- Every page keeps rendering the full legacy template in this commit —
the site stays functional; the split lands in Part 2.
- Tests: four pages 200, wizard 301s + new URLs render, storage-label
redirect Location prefix + flash renders on /storage, wrong-password
inline re-render. Red-proven vs pre-split code (Location was
/settings?..., no 301s).
Found during the D0 click-through: OffboxTarget.LastRun persists as an
RFC3339 string (settings.json), but backups.html passed it to timeAgo
(expects time.Time), so GET /backups returned 500 on any node where an
off-box backup had ever run. Pre-existing since v0.93.0 (NAS Part B),
unrelated to the re-skin. New timeAgoStr funcmap helper parses the
string (falls back to the raw value); also v2-token fix for the offbox
error hint (var(--danger) fallback hex) and the dead stat-running/
stat-stopped classes on the offbox status card.
- settings.html: Aktív/Inaktív rows -> run-blue check / neutral gray
(Inaktív no longer red), update states -> run/crit/progress with
sprite check/x/spinner, pencil + cancel buttons -> icons, storage
badges -> tags, host-disk bar + JS drive capBar -> meters
(usageColorClass -> nominal/warn/crit), state-text-* consumers on the
new suffixes incl. JS-built class names.
- debug.html, app_info.html, storage_init/attach.html, logs.html:
emoji -> sprite icons or plain text in templates AND JS strings.
- catchall.html (standalone, no sprite): v2 token sweep of its inline
style, status emoji -> inline SVGs; a stopped app renders neutral,
not red.
- login.html: two-tone H1 (last word blue-bright).
- setup_hub_versions.html: stale var(--border,#30363d) fallback -> v2.
- Test Group F grep gate: all 34 banned patterns at ZERO across
internal/web + internal/setup; Scenario E test now asserts the
'--bg-0: #0A1220' token literal.
- Vendor Plus Jakarta Sans + JetBrains Mono as variable woff2 (latin +
latin-ext) under internal/web/static/fonts/, embedded via go:embed and
served at /static/fonts/ (font/woff2, immutable cache). Google Fonts
@import replaced with @font-face rules preserving unicode-range —
removes the CDN dependency that silently broke on offline nodes.
- Add templates/icons.html: vendored Lucide sprite (30 icons, symbol
ids i-<name>), included at the top of <body> in layout.html.
- Fix setup wizard handleCSS: serve the embedded web.StyleCSS() instead
of a dataDir-derived filesystem path that never exists in the
container (production setup silently served minimalCSS). Fallback to
minimalCSS only if the embedded read errors, with a WARN log.
- Tests: font route + StyleCSS accessor (web), Scenario E embedded-CSS
test (setup; red-proven against the pre-fix handler).
runStorageInit/runStorageAttach resolved the fs UUID only via agent.Disks(),
which does NOT include a raw (unenrolled, non-PVE-storage) device — so a raw
candidate could be offered but never enrolled ("no fs identifier"). New
resolveEnrollUUID falls back to the raw-device scan (/disks/candidates), which
reports each free disk's durable_id (uuid:<fs-uuid>). Both enroll paths use it;
legacy re-attach (drive in /disks) still works. Test + red-proof.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both wizards now source candidates from the agent's Impl-2a raw-device scan
(GET /disks/candidates, proxied) instead of the Observe-based /api/disks — so a
brand-new non-PVE-storage drive is finally discoverable + enrollable end-to-end.
agentapi.ListCandidates + a passthrough proxy (no controller-side filtering; the
agent's unclaimed filter is authoritative). storage_init renders `initialize`,
storage_attach renders `attach`; the enroll flow + Impl-1 guarded mkfs unchanged.
Tests + go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PushResponse.ConfigVersion from the report ACK; ConfigRefresher reconciles vs.
the last-applied version (settings.applied_config_version) and on a change calls
bootstrap.RefreshConfig (re-pull controller.yaml + re-merge local_api) then
GracefulSelfRestart. First-run records baseline (no restart); unchanged = no-op
(no storm); failed pull keeps config + retries. Companion to hub v0.26.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Live validation surfaced 'option sftp.args is not known' on restic 0.14.0; switch to the
portable sftp.command SSH invocation (ConnectTimeout/StrictHostKeyChecking preserved).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Encrypted restic repo over SFTP for the app-data tier (the off-site 3-2-1 leg). A dead
NAS fails fast via -oConnectTimeout (spike Q8), never hangs the runner; secrets are 0600
files (ride DR via PBS whole-CT); init-if-absent, retention forget --prune, restore,
single-flight, per-app toggle + UI. restic re-added to the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Controller-side of NAS network storage, proxying to agent A1 /netstorage/*. Distinct
'network' storage kind (no drive lifecycle); add/list/remove + per-share health UI;
unreachable NAS is a recoverable warning, never the drive missing/stop cascade; SMB
creds pass through to the agent, never persisted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
A channel broken at startup/reseed (e.g. controller boots into pin_mismatch) was dashboard-only,
no operator email ever. New 'alerted' flag drives alerting instead of prev=='': born-down
non-transient alerts cycle 1; transient still N>=2; healthy first-obs silent; recovery re-arms.
Red-proof + companion included.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
New internal/channelhealth Checker: ~60s probe via the PRODUCTION memoized client
(Server.ProbeAgentChannel, GET /storage), classifies failures (spike Q1 map), debounces transient
reasons (N>=2; construction error latches distinctly), seeds first obs, alerts operator+dashboard on
transition. Notifier.NotifyAgentChannelDown/Recovered (English, operator-only), AlertManager dashboard
banner (Hungarian). No agent/hub change. Spike-proven.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Gap 1: third shim listener :2526, plaintext, does NOT advertise STARTTLS (TLSConfig
nil) — for opportunistic-STARTTLS clients with no cert-skip (cal.com, nextcloud).
Gap 2: SMTPMapping tls_mode (picks port 2525/2526/2465) + from_domain_var (split
local-part + domain for nextcloud's MAIL_FROM_ADDRESS/MAIL_DOMAIN). Default keeps
existing apps on 2525. Hub untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The controller honors an operator-enforced minimum version (FLOOR) on the hub
report ACK and auto-updates to the floor when below it (managed default, no click),
reusing the Phase 1 in-guest-pull + agent-swap + rollback. Latest stays the opt-in
button; the floor is the auto-target, never latest.
- pusher.go: PushResponse += min_controller_version, latest_version (existing ACK seam)
- main.go: OnPushResponse → updater.SetFloor + MaybeAutoUpdate (rides report cycle)
- updater.go: SetFloor/GetFloor + MaybeAutoUpdate reusing performUpdate (auto-floor);
no-op at/above floor, floor>latest, dev/no-agent/backup; no flap (in-mem+persisted)
- settings UI (HU): floor display + auto restart-poll during an auto-update
- tests: below/at/floor>latest/no-flap/raised-floor; below-floor red-proof verified
- no agent change (reuses Phase 1 POST /controller/swap)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
syncFileBrowserMounts no longer force-recreates FileBrowser unconditionally:
captures config.yaml+compose before writes, re-reads final content after, and
recreates only when they actually changed (new pure helper fbNeedsRecreate).
Controller restarts / no-op storage syncs now issue a plain up -d and do NOT
bounce the customer's file UI. Restore-mode DB reset still forces a recreate.
Dockerfile: removed the unused restic apt package (disk-tier restic moved to the
host agent; no controller code execs the binary). ResticSchedule/migrateResticToRsync
config+settings paths untouched (still live in the dashboard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FpBYrZCt9sFDqLgbG5GRGD
In the intermediary-mount model an enrolled drive binds live into the running
guest (no reboot), so the "… meghajtó aktiválásra vár / Újraindítás most" banner
was an obsolete relic — also dead since v0.78 (pendingActivationDrives keyed on
the raw MountPath vs the now-stable sp.Path). Removed the banner block +
activatePendingDrives JS (settings.html), the PendingDrives feed (handlers.go),
and the dead pendingActivationDrives helper + its unused internal/system import.
Renamed handleStorageActivate -> HandleServerReboot (split out a testable
serverReboot core), removed the /api/storage/activate case, and mounted the
handler at the new non-storage route /api/server/reboot (RequireAuth+CsrfProtect).
The agent GuestReboot primitive is reused unchanged.
Added the standalone "Kiszolgáló újraindítása" settings card (sibling to the
controller-only "Vezérlő újraindítása"), reusing the pollRestart() loop.
Test: TestHandleServerReboot_CallsGuestReboot (fake diskAgent asserts GuestReboot
invoked once + 202). diskAgent/mockAgent gained GuestReboot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
The storage card displayed each external drive's raw host PVE mount (/mnt/<name>,
which doesn't exist in the guest) instead of the stable in-guest path
(/mnt/felhom-drives/<name> = guest_path) the registry/HDD_PATH/FileBrowser use.
The eject/wipe buttons also posted the raw path, so they would unmount the drive
but leave the stable registry entry orphaned, and the impact warning found no apps.
Fix: card sub-line + eject/wipe buttons use the stable path (regKey); type-to-confirm
name uses the basename; register keeps the raw path. handleStorageWipe maps to raw
via agentWhere() for the agent eject (matching handleStorageEject). Agent ops
unchanged (same raw paths); display + registry bookkeeping corrected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
Follow-up to v0.78.0. The disk-view JS (settings.html regBadge/actions) checked
registration by the raw mount_path, but the registry stores the stable path since
v0.78.0 — so enrolled drives showed a spurious "Nem regisztrált" badge + Register
button. Fix: regKey(d) = d.guest_path || d.mount_path (agent reports the stable
guest_path per disk); registerDrive still posts the raw path. Display-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
handleStorageRegister (the "Regisztrálás" action for an already-mounted,
unregistered drive) registered the raw /mnt/<name> path verbatim, unlike
runStorageInit/runStorageAttach which map to the stable /mnt/felhom-drives/<name>
path the agent actually binds the drive at. The controller then watched an empty
placeholder dir on the guest rootfs → "Rendszermeghajtón" + stuck "activation
pending" banner after a re-provision.
Fix: register stablePathForName(path.Base(req.Where)); attachIntoGuest still uses
the raw path (the agent operates on raw). Test + red-proof added.
Diagnosis: felhom.eu/documentation/audits/DIAGNOSE-drive-bind-after-reprovision-2026-06-23.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
Belt (ensureUserdataMounts) + FileBrowser sync skip ensure/mount when an external
drive root is not a live mountpoint -> no 'mkdir userdata: permission denied' + no
rootfs-shadow during a drive-absent window. System/local path never gated. Reuses
system.IsMountPoint; matches planDriveGates external-only rule. T1-T4 + red-proofs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
agentClient() built a new agentapi.Client (new bare http.Transport, IdleConnTimeout:0)
per call and discarded it without closing idle conns -> one leaked idle ESTABLISHED
socket per call to the agent :8443, exhausting the ephemeral port range after ~5 days
(EADDRNOTAVAIL). Memoize one shared client via sync.Once; harden Transport
(MaxIdleConns/PerHost + IdleConnTimeout 90s). Agent/firewall untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DR recipe slice (controller half), grounded in SPIKE-dr-recipe-2026-06-16. The
controller emitter is the BOUNDARY enforcement point: v1 ships an explicit
allowlist {catalog_ref, enabled, storage_bindings} and reads NOTHING from
AppConfig.Env, so no ENC:/token/password can leak. storage_bindings parsed from
the compose (${HDD_PATH}/${USERDATA_PATH} volume binds -> {container_path,
drive, subpath}).
Load-bearing tests: TestBuildAppRecipe_NoSecrets (synthetic-secret app -> none
leak) + TestBuildAppRecipe_AllowlistIsLoadBearing (red-proof companion) +
TestAppStorageBindings + TestBuildDRRecipeAppHalf. Red-proofed live: forcing the
emitter to dump Env makes the boundary test fail. recipe_version=1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
processGuestBootChange recreated the drive-backed app stacks but never re-synced
FileBrowser (base-infra, no HDD_PATH), so its drive mounts went stale after a
reboot. Now, AFTER pollLiveBinds confirms the live binds and the apps are
recreated, trigger go s.SyncFileBrowserMounts() so FileBrowser converges against
the now-live drives. Refactored into pure recreateDriveBackedApps(stacks, present,
recreate, syncFB). Tests: FB sync runs once after recreate (red-proofed companion);
runs even when nothing recreated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Found FileBrowser down after the host-reboot drills (my 'all recovered' claim only
checked the 8 felhom-flash apps). Two distinct gaps:
(A) agent-side: felhom-usb did not re-mount after the host-reboot device-letter
swap (mount unit inactive; agent reports present drive as durable-id absent) —
a felhom-agent bug, out of controller scope.
(B) controller-side: FileBrowser is base-infra (no HDD_PATH) so processGuestBootChange
skips it; its SyncFileBrowserMounts runs once at startup, racing the bind, not
retried. Recommended fix: call SyncFileBrowserMounts after the live-bind poll.
Recovered FileBrowser live (started felhom-usb mount unit -> agent bound it ->
restarted controller -> FB Up healthy). felhom-usb userdata was intact, just hidden
behind the unmounted placeholder. Corrected the host-reboot trust caveat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-drilled the HOST reboot (felhom-pve) x2 after the follow-up challenge: both
recover all 8 drive-backed apps automatically; host-btime prefix advances so the
host path triggers processGuestBootChange; the boot-race manifests on host reboots
too (not just guest). Corrected the REPORT's 'host path unaffected' claim.
Regression analysis: the v0.68 sweep genuinely exercised the recovery (it surfaced
a real state-filter bug on the host reboot), so the agent path worked then and
regressed after (controller.yaml reset to golden no-local_api baseline on each
recreate; old MaybeIngest never re-merged). v0.70.0 config round-trip exonerated
(GET /api/config is verbatim). ensureLocalAPI closes the regression on both paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live diagnosis of drive-backed apps stuck Exited after a pct reboot pinned THREE
sub-causes, fixed together (hardening the existing processGuestBootChange, not a
parallel mechanism):
1. Agent-path blocker (live root cause): agentClient() returned "agent not
configured" (cfg.LocalAPI.Endpoint empty), so processGuestBootChange AND the
whole drive gate bailed at the first guard. bootstrap.json had a complete
local_api block, but MaybeIngest returned immediately on "already configured"
so a controller.yaml seeded before local_api existed never got the agent path.
Fix: MaybeIngest now calls ensureLocalAPI on the already-configured path,
merging local_api from bootstrap.json into the existing controller.yaml when
missing (no hub re-pull, config preserved; idempotent + fail-safe).
2. Boot-race readiness gate: processGuestBootChange sampled BoundUnderParent once
during fast startup, racing the ~18s rebind, recreated nothing, burned its
boot-id one-shot. Fix: gate on the REAL live in-guest bind -- driveBindLive
checks /mnt/felhom-drives/<drive> is a mountpoint in the controller's own /mnt
rslave /proc/self/mountinfo; pollLiveBinds waits for it (bounded ~120s) before
recreating via the normal pipeline. shouldRecreateOnBoot stays state-independent
so stuck-Exited create-time-failure apps are included.
3. Single-shot fragility: processGuestBootChange ran only once at startup; a
briefly-unreachable agent right after a guest reboot stranded recovery. Fix:
driveGateLoop runs it every periodic tick too (idempotent, boot-id gated).
Tests (non-hollow, pre-fix companions, red-proofed): pollLiveBinds waits then
reports live / never-live stays absent / single early sample misses; ensureLocalAPI
merges local_api into a configured controller.yaml that lacks it / no-ops when
present. Live-accepted with repeated pct reboot 9201.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sub-cause: on guest pct reboot, in-guest dockerd auto-starts unless-stopped
drive-backed apps ~18s BEFORE the agent re-binds the drive; the create-time
volume bind fails (mkdir /mnt/felhom-drives/<drive>/userdata: permission denied)
and RestartCount=0 means it's never retried -> stuck Exited. The existing
recovery (processGuestBootChange) RAN but raced the rebind: it sampled the
agent's BoundUnderParent once during fast startup (not live yet), recreated
nothing, and persisted the new boot-id -> burned its one-shot. The periodic gate
never recovered them either (first observation after the rebind -> no transition).
Fix (harden the existing mechanism, no parallel one): processGuestBootChange now
gates on the REAL live in-guest bind. driveBindLive checks whether
/mnt/felhom-drives/<drive> is an actual mountpoint in the controller's own /mnt
(rslave) /proc/self/mountinfo -- true only once the agent's bind propagated,
exactly when docker can recreate the app. pollLiveBinds waits for that (bounded
~120s, poll 2s; rebind lands ~18s) and only then recreates via the normal
pipeline, including stuck-Exited create-time-failure apps (shouldRecreateOnBoot
is state-independent). Single-flight; absent-after-window drives left to the gate;
host-reboot path unaffected; guest-only reboot path now covered.
Tests: pollLiveBinds waits through the rebind then reports live (recreate fires);
never-live drive stays absent; pre-fix companion (single early sample misses the
not-yet-live bind). Red-proofed against a no-wait single-sample.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Vezérlő újraindítása card was accidentally placed inside the
{{if .RetrievalPassword}} block, so it was hidden whenever no retrieval
password is set (e.g. the demo). Moved it outside so the self-serve restart
is always available. Found during live verification of v0.70.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause (diagnosed live, not the hypothesised JS throw): filterCountries runs
fine and correctly populates the list, but reveals it with `list.style.display=''`.
The .geo-country-list CSS default is `display:none` (style.css), so clearing the
inline style falls back to none and the populated list never shows — no console
error, just an invisible dropdown. Latent since the geo feature's first commit.
Fix: reveal with 'block'. Verified live (typing "Német" now lists Németország).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Geo settings save and manual geo sync now fire an out-of-band, non-blocking hub
report push (Router.reportPushNow seam, wired in main.go to BuildReport+Push in a
goroutine) so the hub reflects the new geo state / clears a stale last_sync_error
within seconds instead of after the next ~15-min cycle. Scope: geo handlers only.
- builder.go always populates report.GeoRestriction (Enabled=false, empty countries
when nil/disabled) via new buildGeoRestrictionReport helper, so the hub always
renders the geo section ("Inaktív" when off) instead of hiding it via omitempty.
- Tests: geo save success → push once; invalid country → no push (companion);
buildGeoRestrictionReport(nil) → non-nil disabled (companion vs old nil-omit).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /api/config/apply now takes effect via a graceful SELF-RESTART instead of
logging "restart needed" and leaving stale in-process singletons (the CF client
is built once at startup, so a rotated Cloudflare token never applied until a
manual LXC restart). Container is restart:unless-stopped, so a clean os.Exit(0)
auto-restarts with fresh config.
- New gracefulSelfRestart helper behind an injectable Restarter seam (Router.restart
+ SetRestarter) so the exit is unit-testable.
- configApply: no-op guard (byte-identical re-push → no write, no restart), else
write → 200 (flushed) → restart. Removed stale "restart needed" wording.
- Removed the dead OnConfigApplied hook (Phase-1-retired infra-backup push; the
self-restart reloads everything and a fresh report is pushed on startup).
- New POST /api/selfrestart (auth+CSRF via /api/ mount) + "Vezérlő újraindítása"
settings button: confirm → POST → poll GET / every 2s → reload.
- Tests: changed→restart once; identical→not called (companion); invalid→not called;
selfrestart→restart once.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Controller half of Phase-1 Infra Backup retirement (hub v0.12.0;
SPIKE-infra-backup-2026-06-15). Pure dead-code removal, no behaviour change.
- Remove Pusher.PushInfraBackup (caller-less; hub endpoint gone).
- Remove Notifier.NotifyBackupCompleted / backup_completed event (caller-less
since slice 8C; hub deadline check now reads agent host-report PBS snapshots).
- Remove report.BackupReport.ResticPassword — builder never sets it post-8C
(confirmed in source + live), but it historically leaked the restic password
into the hub's plaintext reports store.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The migration journal keeps returning the last completed job; the resume-view
watched any job and migWatch's done-branch reloads the page -> endless reload
loop after any migration. Resume-view now watches only in-progress jobs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.stack-title-row gains flex:1+min-width:0 so it shrinks/wraps; .stack-state-badge
gains flex-shrink:0 so the nowrap badge is never compressed. Only unhealthy cards
clipped (the route-unpublished warning inflated the title-row).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
E1 caught it: filtering on State!=stopped missed apps docker hadn't auto-restarted
yet at the one-shot instant (5 apps exited after host reboot). Now recreates every
deployed present drive-backed app regardless of state (deployed=should run).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recreateBootStaleApps ran before the stack manager finished scanning (GetStacks
empty) so it found no apps; add a bounded wait for stacks before the one-time
boot-stale recreate. Deterministic guest-reboot convergence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recreateBootStaleApps recreates a present drive-backed app when boot-stale OR
exited/restarting/unhealthy (the recency-only gate missed already-exited apps).
Still skips healthy long-running + cleanly user-stopped apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes guest-reboot convergence. driveGateLoop runs recreateBootStaleApps once
at startup: deployed drive-backed apps whose drive is present (BoundUnderParent)
and whose containers started recently (fresh guest boot, not a controller-only
restart) are recreated (down+up) onto the re-propagated drive. Paired with agent
v0.35.0's drive re-propagation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The drive-absent gate treats a stable path usable only when bound under the parent
(BoundUnderParent), not merely host-mounted. Makes a host reboot converge: apps
stay gated until the agent binds the drive under the parent, then are restarted
(recreated) on the populated path. Test updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix caught live: planDriveGates falsely marked the internal SSD path
/mnt/sys_drive/felhom-data disconnected (agent never reports it), which would
block starting SSD-resident apps. Gate now skips non-/mnt/felhom-drives/ paths.
Regression case added. No apps were stopped (none depended on the SSD path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drives are visible in-guest only at the STABLE /mnt/felhom-drives/<name>; the
registered path + HDD_PATH + FileBrowser source repoint there while agent calls
map back to raw /mnt/<name> (agentWhere). Enroll binds-under-parent before
register. Drive-absent GATE (planDriveGates + 30s driveGateLoop) stops/blocks
apps when a drive vanishes and auto-restarts on return; start-gate refuses start
when the drive is absent. H1 endpoints (disconnect/reconnect/restart-apps) routed
onto host-side ops. Non-hollow tests + companions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gtstef/filebrowser is a single Go binary that ignores a UMASK env (verified live:
-e UMASK=002 leaves PID1 0022), so RenderFileBrowserCompose wraps the entrypoint
sh -c 'umask 002; exec /home/filebrowser/filebrowser'. Customer-created folders now
come out 2775 (group-writable) so group-1000 apps can write into them. Test asserts
the wrapper is rendered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DeployStack's initial compose-up builds env from deploy values (not stackEnv), so
v0.66.0 missed USERDATA_PATH on first deploy → ${USERDATA_PATH} resolved to '' and
Docker bound a root-owned dir at the container root (found live: radarr /media/movies
was 0:0 755). Shared withUserdataPath injector now used by stackEnv AND
composeExecWithEnv. Regression test included.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
agentapi.Decommission + handleStorageDecommission (migrate-all-or-none, Change 2):
migrate-then-decommission via the migration done-hook, or decommission-anyway (stop
apps, keep HDD_PATH). 'Hiányzó tárhely' badge on dashboard/stacks/app card when an
app's drive is decommissioned/disconnected/absent. Change 4: registerStoragePath
clears the decommissioned marker on re-enroll (ClearDecommissioned had no callers).
Non-hollow tests incl. mutation-proven Change-4 companion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ServeStorageAPI gains POST /api/storage/migrate (whole-namespace), POST
/api/storage/migrate-app (single app), GET /api/storage/migrate/status (poll).
settings.html: the greyed migrate-all span becomes a real target-select + button +
shared progress panel; app_info.html gains a per-app 'Áthelyezés másik tárhelyre'
control. Both poll the shared status endpoint and render Hungarian phase progress.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
internal/stacks/migrate.go: crash-safe, resumable namespace migration over the
controller's /mnt RW mount. Two entry points (whole-namespace + per-app) share one
journaled pipeline: validate -> stop -> copy (rsync -a --checksum, additive; conflict-
merge walk for non-app content) -> verify -> flip+redeploy (RedeployFromEnv) -> cleanup.
CLEANUP (the only destructive step) is gated on all units verified AND all apps
redeployed. Single-flight; mutual exclusion with the backup orchestrator (Change 3).
Non-hollow tests incl. mutation-proven collision + cleanup-gate companions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the F9 storage episode — live validation of a user-facing feature
must exercise the real flow end-to-end, not shortcut via direct
API/agent/CLI calls. Low-level mechanism tests are exempt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A1: AutoDiscoverStoragePaths no longer bails on a non-empty registry;
registers only deployed-app paths missing from the registry. Never
mutates/removes existing entries, never re-adds or reactivates a path
present in ANY state (incl. Decommissioned), never flips IsDefault.
A2: InferStorageLabel maps base==felhom-data namespace dir to
'Belső SSD (rendszer)' to disambiguate the internal system volume.
Table-driven tests incl. a companion that fails without the
skip-by-presence guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent (v0.31.0) now returns these on /disks, but the controller dropped them when
re-marshalling into its agentapi.DiskInfo mirror. Added both fields (additive) so they reach
/api/disks + the dashboard: wipe_durable_id (gate scheme, for the wipe-confirm) and guest_attached
(drive bound into THIS guest vs merely host-present). Controller behaviour otherwise unchanged.
ListDumpFiles ran ValidateDump (line-by-line scan) for every dump on every ~5-min
RefreshCache cycle — wasted I/O+CPU on large customer dumps. ListDumpFiles now takes
an optional cached(name,size,mod) lookup; on a (size+modtime) match it reuses the
prior result and skips ValidateDump. settings.DBValidationCache gains Size+ModTime;
listAllDumpFiles builds the lookup from the persisted cache and writes back only fresh
validations (cache miss), so an unchanged dump triggers neither a re-validation nor a
settings.json write each cycle. nil cached = legacy validate-always (back-compat).
Tests: cache-hit skips validate (sentinel), cache-miss validates, nil validates.
deriveStackName pure-suffix-stripped on '-' (postgres/db/mariadb/.../cache), so a
stack whose slug ENDS in a role token (e.g. 'my-cache') was misattributed (stripped
to 'my') — filing its DB dump under the wrong/nonexistent stack. Now threads the set
of deployed stack names (m.knownStackNames() <- ListDeployedStacks) into
DiscoverDatabases and cross-references: candidate suffix-strip if known, else the
container name if it IS a known stack, else longest known stack that is a prefix
(handles <stack>_postgres / <stack>-1), else legacy strip. nil/empty known = legacy
behaviour (appexport passes nil). Table test incl. the my-cache case (fails pre-fix).
All shippable work → main directly; report-only artifacts → felhom.eu/documentation/
(audits/backlog); risky/supervised fixes implemented on main during the supervised
session, not prepared on a branch; unattended escape hatch = revert+report, never park
on a branch. Supersedes the old 'prepared on fix/... branch, pending review' pattern.
The live-drive findings + fixspec now live in felhom.eu/documentation/audits/
(with the other audit records); they no longer belong loose at the controller
repo root. Transient SESSION-*.md working logs dropped (their lasting content is
in CHANGELOG/CONTEXT and the audit records). Trunk-based no-branches cleanup.
The cgroup-only approach was a no-op on the demo: the controller container's OWN
cgroup is unlimited (the 2GB cap is on the LXC ancestor, hidden), and /proc has no
lxcfs, so it kept reporting the host's 16GB. The Docker daemon runs IN the LXC, so
'docker info' MemTotal reports the guest's real cap (2048MB) — now the authoritative
source (cgroup limit preferred when present, e.g. non-nested). The deploy memory
guard now uses the controller's committed-app memory (sum of running mem requests)
for 'used' — accurate and cheap — instead of host /proc RSS (unobservable per guest;
would make the guard never/always fire). /api/system/info reports the guest cap as
total and committed memory as used. Tests: cgroup-limit path, docker-info fallback
(nested case), GuestMemTotalMB fallback (dockerMemTotalFn stub).
The restore paths (RestoreFromRecoveryUnit + the RestoreApp fallback) repopulated
Docker volume tars but NEVER replayed the captured <stack>-<dbtype>.sql dump, so
DB-resident data (e.g. rows in a DB whose data dir is a bind mount) did not come
back — the romm marker round-trip in the audit lost the row.
New appbackup.ImportDump (read-side counterpart to DumpOne) replays a .sql/.sql.gz
into the running DB using the live container's OWN discovered credentials (no env
threading; reuses DiscoveredDB + getMariaDBPassword). backup.reimportDBDumps
orchestrates it AFTER volume restore + stack bring-up, so the logical dump WINS
over any volume-tar copy of the DB (operator-chosen precedence). pg_dump
--clean --if-exists and mariadb-dump (default --add-drop-table) make replay
idempotent; psql ON_ERROR_STOP=1 surfaces real import errors.
Also: volume-restore per-volume failures and DB-import failures now SURFACE (the
restore returns an error) instead of a swallowed WARN, so a failed data restore
cannot read as success.
Tests (restore_db_test.go, injectable discover/import seams): imports when dump+DB
present, failure surfaces, no-dump skips discovery, dump-but-no-matching-DB is a
non-fatal skip. Live DB round-trip to be validated post-deploy.
Traefik only publishes a route to a healthy container, so an unhealthy deployed app
returns 404 at its URL though the container runs — previously shown only as 'Nem
egészséges' with no hint the URL is dead. New routeUnpublished() funcmap helper +
a distinct indicator on the dashboard and stacks cards (gated on .Deployed). Tests:
routeUnpublished across all states, real templateFS parses with the funcmap, and the
card guard renders the indicator only for deployed+unhealthy.
The Hub config-apply handler wrote controller.yaml 0644; it holds cf_api_token,
cf_tunnel_token and hub api_key in plaintext. New writeConfig0600 helper writes
0600 atomically (tmp+rename, bind-mount fallback) and chmods to enforce 0600 even
when the file pre-existed 0644 (os.WriteFile doesn't chmod existing files).
Test asserts mode 0600 (Linux; skipped on Windows). Setup path already used 0600.
The dashboard stacks list served the in-memory map refreshed by a 30s ticker, so
container state lagged Docker health by up to ~30s after a deploy. RefreshStatus
is a cheap docker-ps refresh; 10s (matching health-probes) cuts the lag without
loading Docker.
F4: GET /api/stacks/rescan fell through to GET /stacks/{name} → misleading
'stack not found: rescan'. Now returns 405 + Allow: POST.
F6: the deploy POST returns before compose/health complete (async; UI polls).
Message changed 'Stack X deployed' → 'Telepítés elindítva…' and status 200→202
Accepted, so API/script consumers aren't told a deploy finished when it hasn't.
UI checks data.ok (not HTTP status), so 202 is safe.
A failed agent format (e.g. 502 'device is mounted', ok:false, data:null) fell
through FormatDisk's trailing 'return out, nil', so the web layer reported a
zero-value FormatResult as ok:true — a failed DESTRUCTIVE format read as success.
postWithStatus now returns the full envelope; FormatDisk returns a non-nil error
on any non-2xx/ok:false that is not a recognized refusal (403/needs-confirmation).
Test TestFormat_MountedFailureSurfacesError (502 → non-nil err) fails on old code.
/api/system/info reported the Proxmox host's 16GB (the controller container
reads host /proc/meminfo with no lxcfs), defeating the deploy memory-headroom
hard-block (deploy.go uses GetMemoryMB). readMemInfo now prefers the cgroup
memory limit (v2 memory.max / v1 memory.limit_in_bytes; sentinels = unlimited)
when finite and below the host total; used = memory.current/usage_in_bytes.
Test info_cgroup_test.go (cgroup v2 cap wins, v2 max sentinel, v1 unlimited,
v1 finite) — fails on pre-fix code.
CLAUDE.md line 31 wrongly said 'bulk strip has NOT happened' — slice 8C executed
the de-privileging (storage/restic/watchdog deleted; disk ops via agentapi).
Corrected to current state + pointer to central docs. CONTEXT banner -> v0.60.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NewServer launches the SyncFileBrowserMounts goroutine (reads integrationMgr)
from the constructor, BEFORE main.go's SetIntegrationManager write — so the
init-only happens-before that covers the other Set* fields does NOT hold here,
making it a genuine data race (handlers.go:358/360/1433 reads vs server.go:162
write). Converted the field to atomic.Pointer[integrations.Manager]; setter
Stores, all 3 readers Load(). Regression test reproduces the concurrent access
(clean under -race; flags on the pre-fix plain-pointer field).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- controller/README.md: authoritative banner (v0.59.0, docs moved to
felhom.eu/documentation/controller/, bootstrap-deploy note); legacy body retained.
- CONTEXT.md: banner refreshed to v0.59.0 state.
- Removed controller/mnt/user-data/outputs/... — an accidental Claude-sandbox
output dir (stale duplicate README) committed in the initial import.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single write (SetStackProvider, main.go:225) was mutex-guarded while all 11
reads were unlocked — the lock implied a runtime concurrency the reads don't
honour. It is called once during single-threaded startup before any goroutine,
so the write happens-before every read and no race exists. Removed the
misleading lock and documented the init-only contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deploy.go, one slice (both edit SaveAppConfig / the deploy goroutine):
CTRL-T2-1 (ghost-deployed on crash): DeployStack wrote app.yaml Deployed:true to
disk BEFORE the async 'docker compose up -d'; a crash during the image-pull
window left a ghost-deployed stack (Deployed:true, no containers) that DeployStack
then refused to redeploy. Now the env is persisted with Deployed:false
(transitional), and Deployed:true is written by runComposeDeploy ONLY after up -d
succeeds. In-memory Deployed stays true during the pull to preserve the
no-stale-Telepítés-button UX. On a post-success save failure, revert so the stack
is redeployable.
H10 (plaintext secret on encrypt failure): SaveAppConfig logged a WARN then fell
through to persist the secret in PLAINTEXT. Now fail-closed: return an error on
crypto.Encrypt failure, never write plaintext. Callers already propagate it.
Regression tests: H10 fail-closed (+ good-key encrypts) and the CTRL-T2-1
transitional durable-state contract (transitional reads not-deployed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
manifest.AppName / HDDSubdirs / VolumeNames are attacker-controlled JSON inside
an imported .fab and reach filepath.Join+MkdirAll/extractTar with a trusted base
(restore.go:339/606/678). UnmarshalManifest did zero validation, so '../..' in
any of them escaped the stacks / HDD destination dir.
- New appexport.ValidateSegment + validateManifestPaths; UnmarshalManifest now
fails the parse on a traversal segment (the chokepoint).
- Defence-in-depth ValidateSegment guards at the HDD-subdir and volume-name join
loops in restore.go.
- ConfigFiles deliberately NOT validated (holds dotfiles like .felhom.yml; never
used in a restore join).
- Permanent regression test (was the deep-sweep failing audit test) now asserts
rejection of traversal + acceptance of legit names.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reserved-buffer headroom guard on the Docker-data volume (system/dockervol.go,
max(5GB,10%)); deploy-time hard gate refuses (HTTP 507) when below the buffer
(api/router.go); deploy page warns + disables the button (deploy.html); runtime
disk monitor confirmed to watch the Docker volume above the buffer. Log rotation
baked into the golden (agent side). Phase 1 = felhom-agent v0.29.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Part A of the UI-fixes/storage-spike spec.
A1: enrichHostStorageTargets sorts /api/host-metrics storage_targets
server-side and attaches friendly Hungarian labels + purpose, fixing the
#host-storage-bars reorder-on-poll bug. Display labels only — PVE storage
ids are never renamed.
A2: new GET/POST /stacks/{name}/backup Tier-2 config panel; the "2. mentés"
Beállítás button is repointed there from the dead-end deploy page. Customer
can pin a target drive or disable Tier 2; preference is preserved across the
runner's status writes. Always visible (single-SSD + non-HDD apps included).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4A: scope FileBrowser bind to <drive>/appdata (recovery units + Tier 2 copies under
backups/ are no longer mounted into FileBrowser — customer can't browse/delete the
thing that restores them). 4B: deploy storage-selection step states the chosen drive
holds files while the DB runs on the fast internal SSD + is backed up with the app.
4C: buildStorageBars stable sort + purpose description on the monitoring storage list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tier 2 rsync-mirrors each HDD app's recovery unit + appdata to a DIFFERENT physical
disk (the only off-drive protection bind-mounted userdata can get; PBS can't reach it).
Auto-enabled, auto-target: prefer another registered drive (different physical disk via
system.SamePhysicalDevice), else the internal SSD for SMALL units only — with a
size-aware headroom guard that REFUSES rather than fill the ~8G guest rootfs, recording
an honest "needs 2nd HDD" status. Status persisted via the surviving CrossDriveBackup;
"2. mentés" UI card now populated. Daily tier2-backup job + POST /api/backup/tier2.
- backup/tier2.go (engine+selection+headroom), tier2_test.go (headroom arithmetic)
- system.SamePhysicalDevice (linux Stat_t.Dev + stub)
- handlers.go Tier2 UI population + tier2DestLabel; backups.html honest no-target reason
- fixed stale TestBackupCopiesOnPath (old felhom-data layout -> in-guest layout)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Demo has no dashboard password (API open: auth+CSRF both skip in that mode), driven
via the public URL. AdventureLog's unit manifest carries data_key_env_vars=[SECRET_KEY]
(catalog->manifest live); with SECRET_KEY unrecoverable, POST /backup/restore REFUSED
with the exact fail-closed message before any compose-up. Full deploy-with-data e2e
blocked by the 8G guest rootfs (AdventureLog images too big — the Phase 3 concern, live).
CHANGELOG/REPORT/CONTEXT updated; demo left clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an in-process orchestration test for RestoreFromRecoveryUnit: success path
calls recreate with non-secret env + recovered secrets merged; data-key-missing
path is REFUSED and recreate is never called. Makes Manager.isDebug nil-safe
(behavior-neutral in prod; cfg is always set) so the gate/orchestration are testable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restore recreates an app from its on-drive unit + the guest's own secrets,
regenerating nothing. reconcileRestoreSecrets (pure, unit-tested) merges the unit's
non-secret env with secrets recovered from the live app.yaml and FAILS CLOSED if a
data-encrypting key is unrecoverable (refuse — a PBS whole-guest restore is needed —
rather than regenerate and corrupt). Resettable secrets missing → warn + proceed.
- backup: RestoreFromRecoveryUnit (manifest -> recover secrets -> gate -> restore
volumes -> recreate definition + redeploy w/ re-pull); falls back to volume-only.
- seams: RecoverStackSecrets/RecreateStackFromUnit (adapter +encKey),
stacks.RedeployFromEnv. Wired into /backup/restore.
- tests: gate (refuse/proceed/verbatim) + data_key parsing.
Gate + reconcile + data_key parsing unit-tested; capture live-validated (v0.53.1).
Full readable-data e2e vs AdventureLog needs the auth-gated dashboard restore — pending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CaptureRecoveryUnit now builds content in memory and skips writes when the unit
is already current (checksum + dump-set + version), so it can run from RefreshCache
(startup + every 5m) without thrashing the USB drive. Units now exist shortly after
startup and track config changes without waiting for the daily DB dump. +idempotency test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
REPORT.md overwritten with the Phase-1 gate run (catalog template fix + agreement
test + live RomM migration on guest 9201, gate PASSED). CONTEXT.md dated entry.
README HDD_PATH/felhom-data convention note corrected for Model-A single-nesting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deploy-side double-nest fix lives in the app catalog (templates dropped the
extra felhom-data segment). This adds the controller-side invariant test that
ties the deploy path (ParseComposeHDDMounts) to the backup path
(AppDataDir/NamespaceRoot) so they can't drift again, plus the v0.52.0 CHANGELOG.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Backups page: whole-guest backup shown as real DR — target label "Biztonsági szerver –
külön hardver (PBS)"; app-data "Távoli mentés" card now reflects the PBS offsite tier
(guestBackupView.Offsite) instead of "nincs beállítva".
- Model-A double-nest fix: appbackup path helpers take a felhom-data NAMESPACE ROOT (no
internal felhom-data join); backup.Manager.namespaceRoot/AppNamespaceRoot resolve
HDD-vs-systemDataPath provenance so a drive-resident app's backups land single-nested
(<drive>/backups/... on the guest = <drive>/felhom-data/backups/... on the host) instead
of .../felhom-data/felhom-data/.... Writes, deletion (GetStackBackupData/RemoveStack/
ProtectedHDDPaths), wipe-warning scan, and export updated coherently; legacy double-nest
dirs kept protected. New appbackup test asserts no doubled segment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4A: user-data drives are backup-target-eligible (not role-locked) — surfaced in
the drive purpose note. 4B: handleStorageImpact returns backup_copies (apps whose
cross-drive backups live on the drive, via backupCopiesOnPath); the wipe/eject
modal warns they'd be destroyed (stays customer-confirmable — copies redundant).
Cross-drive backup engine remains out of scope. Test: TestBackupCopiesOnPath.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pendingActivationDrives() flags registered drives the agent shows attached but not
live-mounted in the container; settings banner + "Újraindítás most" button →
/api/storage/activate → agentapi.GuestReboot. Batches all pending into one restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
agentapi GuestAttach(where) → POST /disks/guest-attach; runStorageInit/Attach +
handleStorageRegister call attachIntoGuest after register (best-effort, P3 heals).
Closes Branch A: enrolled drives become usable in the guest, banner clears.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Part 2 of the USB/backup spec. agentapi: StatusResponse.Backup record, DueResponse
age_seconds, RestoreTestStatus(). New "Rendszermentés (teljes mentés)" section
(read-only: last backup/target PBS-vs-local/next-due/restore-test) + "Mentés most"
manual trigger that goes through the quiesce loop (controller owns quiescing):
quiesce.Loop gains mutex + TriggerNow() (single-flight, async). New
/api/guest-backup/{trigger,status} (distinct from apiRouter's /api/backup/*).
App-data rows relabeled under an "Alkalmazás-mentések" divider. Config → slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
backups.html still referenced .Backup.{RepoStats,LastBackup,ResticSchedule,
NextBackup,PruneSchedule,Retention,SnapshotHistory,LastCheckTime,LastCheckOK} —
fields removed from FullBackupStatus in the 8C de-privileging (disk-tier backup
moved to the agent). Field access on the slimmed struct 500s. Removed the dead
restic/snapshot/repo-stat sections; kept the app-data (DB dumps + per-app) view.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Empirically (staging on 9201): traefik v3 issues a cert from a router-level
tls.domains but NOT from the entrypoint http.tls.domains. So the wildcard moves
to RenderControllerRoute (the always-present anchor): when DNS-01 ACME is
configured it carries tls.certResolver+domains *.<domain>+apex, and every other
router serves that wildcard by SNI (no per-app labels). Reverts v0.42.0's dead
entrypoint-domains + TraefikData.Domain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
traefik's websecure entrypoint now declares http.tls.domains *.<domain>+apex so
it proactively obtains the wildcard via Cloudflare DNS-01 at startup (cert ready
before first client, every router serves it by SNI). Gated on CFAPIToken (DNS-01).
TraefikData gains Domain; ensureTraefik wires cfg.Customer.Domain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
containerOnNetwork misread the absent-key '<nil>' as "already attached", so
wireController skipped docker network connect -> traefik 502'd felhom.<domain>.
Now lists network names and matches exactly. Also removed dashboard.html's dead
CrossDrive* block (slice-8C leftover) that 500'd the dashboard via gt <nil> 0,
exposed once v0.41.1 made the dashboard reachable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EnsureBaseStack now writes a traefik file-provider route
(Host(felhom.<domain>) -> http://felhom-controller:8080) and joins the
controller to traefik-public. Done post-pull (domain known) and idempotently
(write-if-changed + skip-if-connected), so felhom.<domain> reaches the
controller. Completes the v0.41.0 base-infra bring-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New internal/infra package renders traefik/cloudflared/filebrowser from config
(pinned images, single source of truth; web filebrowser path delegates here).
stacks.EnsureBaseStack deploys the traefik-public network + the three stacks,
single-flight + idempotent + non-fatal; wired to first boot and every health
tick. monitor.EffectiveProtected drops cloudflared when no tunnel token.
Section-G fix lives in felhom-agent build-golden.sh (same-path stacks bind).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix the onboarding 401: instead of seeding controller.yaml from the agent's
HOST hub key (which the hub's customer-scoped /api/v1/report rejects), the
controller now PULLS its full controller.yaml from the hub on first boot using
the bootstrap's retrieval passphrase (yielding the customer-scoped key) and
MERGES in the per-guest local_api block.
- internal/bootstrap: contract v1->v2 (customer.id + hub.url +
hub.retrieval_password + local_api; drop host key/identity). MaybeIngest gains
an injected PullFunc (keeps bootstrap free of the heavy report package),
pulls with bounded transient-only retry, merges local_api at YAML-map level
(preserves all hub-emitted fields), idempotent + fail-safe + never-crash.
- main.go: wire report.PullConfig as the pull adapter (maps ErrHubUnreachable
-> ErrPullTransient; auth/not-found permanent).
- Lockstep with felhom-agent v0.19.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove five orphaned HTML templates left behind when slice 8C retired the
disk/storage/restore web handlers (storage_handlers.go, handler_restore.go and
the /api/storage/* + /api/restore/* routes): storage_init, storage_attach,
migrate, migrate_drive, restore. Zero .go references, zero cross-template
references, no route, no nav entry; embed is a glob so deletion is safe (14
templates remain, build + tests green). No behaviour change; the deleted pages
were already unreachable.
Also ships the live demo validation (v0.39.0) writeup in REPORT.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add agentapi HostMetrics() + a thin /api/host-metrics proxy to the agent's
new GET /host/metrics, and a 'Szerver allapota (gazdagep)' card on the
monitoring page rendering host CPU%/load/mem/CPU-temp(n/a)/uptime + per-
storage capacity bars (thin-pool fill, disk temp/wear). Polls every 8s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Quiesce loop resumes (StartStack + clear marker) at the snapshotted phase
instead of done -> downtime whole-backup -> until-snapshot, no consistency loss.
Keeps polling to done/failed (no overlapping backup; post-snapshot failure
observed). Stop-mode fallback to done + crash-safety preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dropped privileged:true + /mnt rshared + /sys + /dev + /etc/fstab + /run/udev
from the bare-metal compose template (controller no longer does disk ops). The
golden bootstrap run was already minimal (8A). Slice 8 CLOSED on the controller.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reflow removes hard mid-paragraph line wraps (code blocks and tables untouched);
rendered output unchanged. Adds the uniform CHANGELOG (cumulative) / REPORT
(overwrite-latest) convention plus a no-secrets rule. Docs/meta only, no version bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Repo renamed on Gitea (admin/deploy-felhom-compose -> admin/felhom-controller).
Updates clone URLs, clone dirs, the customer bootstrap URL, build.sh, BUILDING.md,
README.md, CLAUDE.md, CONTEXT.md and TASK.md to the new name. No functional change:
Go module path and Docker image path (both already 'felhom-controller') untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract the stateless, keep-side app-data backup primitives out of
internal/backup/ into a new self-contained internal/appbackup/ package:
- dbdump.go: DB dump discovery/execution (DiscoverDatabases, DumpOne, ...)
- appdata.go: StackDataProvider + app-data/volume discovery, HumanizeBytes
- paths.go: keep-side path helpers (AppDBDumpPath, AppVolumeDumpPath, AppDataDir)
backup/ keeps every name available via type/const aliases + one-line function
forwarders (appbackup_bridge.go), so the still-present delete-side code
(restic, cross-drive, drive-mount) and the both-side consumers (web/api/report)
compile unchanged. The keep-only consumers appexport and storage are rewired to
import appbackup directly and no longer import backup.
This is the Part-2 prerequisite for the Proxmox port: appbackup has zero
references to restic/cross-drive/drive-mount and does not import backup, so the
delete-side can later be removed without breaking app-data backup or appexport.
Behaviour-preserving: pure move + import/qualifier rewrites, no logic edits.
The four Manager methods (RunDBDumps/DumpAppVolumes/DumpAppVolumesSafe share the
delete-side mutex/status state; RestoreAppFromTier2 reads the cross-drive mirror)
intentionally stay on Manager and delegate to appbackup — for the re-platform step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The backups page template references .HasVolumeData on the status table
rows but the AppBackupRow struct was missing this field, causing a
template error.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Docker named volume backup to Tier 1 (dump to tar, include in restic)
and Tier 2 (copy tars to rsync mirror _volumes/ dir)
- Fix volume name resolution: use project-prefixed names (mealie_mealie_data)
- Fix double Tier 1 in restore dropdown: filter snapshots by app's home drive
- Add Tier 2 restore: RestoreAppFromTier2() restores from rsync mirror
- Show Tier 2 entry in restore dropdown when cross-drive backup succeeded
- Add .fab import link in restore section
- Volume-aware restore type banners and backup content labels
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Users couldn't find metadata provider fields (IGDB, ScreenScraper, etc.)
on the app info page. Move them to the deploy page where all other
settings (integrations, geo-restriction) already live.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Users couldn't find metadata provider fields (IGDB, ScreenScraper, etc.)
on the app info page. Move them to the deploy page where all other
settings (integrations, geo-restriction) already live.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The gtstef/filebrowser image bakes FILEBROWSER_CONFIG=/home/filebrowser/data/config.yaml,
but controller mounts config at /home/filebrowser/config.yaml. Override the env var in both
generateFileBrowserCompose() and docker-setup.sh so FileBrowser reads the controller-managed
config with proper sources and database path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detect and offer to format empty (no filesystem) partitions on the system
disk. Adds IsSystemPartition() for granular per-partition safety checks
instead of blocking the entire system disk. Init wizard shows formatable
partitions with appropriate warnings. Add felhotest demo node to docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deactivated drives (Schedulable=false) now treated like disconnected for
Tier2 backups. New IsStoragePathSchedulable() checks active+connected+not
decommissioned. UI shows yellow "Cél meghajtó inaktív" badge, scheduler
skips silently with WARN log.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, removing a storage drive from the controller only marked it as
disconnected if the StoragePath entry still existed with Disconnected:true.
Drives removed entirely from storage_paths were invisible to the check,
causing Tier2 backup UI to show green "Sikeres" and scheduler to attempt
backups to a no-longer-managed destination.
New IsStoragePathKnown() method covers both cases. UI shows yellow
"Cél meghajtó leválasztva" and scheduler skips silently.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- IsUSBDevice/diskModel: strip findmnt bind-mount suffix [/subdir] before
parsing device path (fixes USB badge not showing for attach-wizard drives)
- crossdrive.go: skip disconnected src/dest drives with WARN log instead of
returning error (prevents noisy error status in settings.json)
- handlers.go: detect Tier2 destination disconnection, set yellow status dot
instead of red, skip ValidateDestination for disconnected paths
- backups.html: new template branch showing "Cél meghajtó leválasztva" badge
with grayed-out info and hidden "Futtatás most" button
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the felhom-controller container as a special entry in the
app_telemetry array sent to the hub. This reuses all existing hub
infrastructure (storage, aggregation, UI) with zero hub-side changes.
The controller's memory/CPU metrics and log warnings/errors are now
collected alongside app telemetry, giving the hub visibility into
controller health, memory trends, and known issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add structured operational logging at INFO, WARN, and ERROR levels to
every controller module. Standardize custom prefixes ([GEO], [SCHED],
[SYNC]) to use [INFO/WARN/ERROR] [module] format. Fix misleveled logs
(WARN->ERROR for data loss scenarios, WARN->INFO for routine operations).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add detailed [DEBUG] logging to every controller module when
logging.level is set to "debug". Each module with stateful debug
uses SetDebug(bool) wired from main.go. Covers stacks, backup,
cloudflare, integrations, system, monitor, settings, scheduler,
web handlers, storage, metrics, API, selfupdate, and assets.
Also includes the app export/import (.fab bundles) feature from
v0.32.0 and its debug page integration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Normal storage add/remove no longer nukes the FileBrowser database volume.
A .fb-reset flag file is written during restore and consumed on next startup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
FileBrowser Quantum caches user source preferences in its SQLite
database. After a restore, the config.yaml gets correct sources but
the database still references the old "srv" source from docker-setup.sh
initial install. Now SyncFileBrowserMounts() detects when sources
changed and runs docker compose down -v to reset the database before
recreating.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After restore, the setup server (port 8081) exits and the main
controller restarts on port 8080. waitForRestart() now polls port
8080 using no-cors mode and redirects there when it responds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The exec > >(tee ...) process substitution has a race condition where
the main shell exits before tee finishes printing. The print_summary
output was written to the log file but never displayed on terminal.
Added sleep 0.5 to let tee flush.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restore flow now calls MountDrivesFromLayout() after writing config,
which mounts drives by UUID and adds fstab entries. Previously drives
from the infra backup were never mounted, causing "Adattároló nem
elérhető" warnings.
Post-restore redirect now polls until the controller responds instead
of using a fixed 5-second timeout that was too short for container
restart.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hub: GFS retention (7d/4w/3m, ~14 versions) in new infra_backup_versions
table. Recovery endpoint supports ?version=ID. New /versions API endpoint.
Dashboard shows backup history.
Controller: local drive backups rotated into history/ (last 5 versions).
Setup wizard shows version picker for Hub restores when multiple versions
exist. Scan results enriched with app names, disk count, history badge.
Local restore supports historical versions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename fails with EBUSY on Docker bind-mounted files (e.g. controller.yaml).
Fall back to os.WriteFile when os.Rename fails.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both autoProcessHubRestore and processHubRestore rendered the progress
page (setup_restore_exec) without starting the executeHubRestore()
goroutine, causing the template to poll forever showing "Indítás...".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docker-setup.sh --hub-customer now generates a minimal controller.yaml
(no customer.id) instead of installing full hub config, triggering the
setup wizard on first run. Hub credentials are passed via env vars
(FELHOM_SETUP_CUSTOMER_ID, FELHOM_SETUP_PASSWORD) so the wizard
auto-fills and auto-processes Hub API calls.
Welcome page shows three options in hub mode: restore from Hub (primary),
restore from local drives, or fresh install. On error, falls back to
manual form with error displayed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace traffic light colors (green/yellow/red) with brand palette:
- Primary actions: blue gradient
- Secondary actions: ghost/outline
- Destructive actions: ghost with red hover (modals keep filled red)
- Running cards: blue glow instead of green border
- Bottom-aligned buttons via flexbox column layout
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
StorageUrl was missing trailing slash — NC's OO connector does string
replacement of server URL (ending with /) with StorageUrl, so without
trailing slash "apps/" merges into hostname producing "nextcloudapps".
Also add "nextcloud" to NC trusted_domains so OO Document Server's
internal callbacks (Host: nextcloud) are not rejected.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers architecture, state management, full lifecycle (6 steps),
both handlers with detailed occ commands and config patching,
ReapplyConfigForTarget, force-recreate rationale, Traefik middleware
for OO HTTPS proxy, UI on deploy page, wiring in main.go, and
corrected API JSON field name (enabled not enable).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SyncFileBrowserMounts: use --force-recreate so FB always picks up
config.yaml changes (bind mount not detected by docker compose up)
- OnlyOffice compose template: add Traefik middleware to forward
X-Forwarded-Proto=https (fixes mixed content errors in browser)
- Nextcloud handler: add StorageUrl=http://nextcloud for internal
file download callbacks from OO Document Server
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SyncFileBrowserMounts regenerates config.yaml from scratch, overwriting
any integration config. The old approach used an async OnStackStart hook
after container restart, which failed due to timing issues (stack state
not yet refreshed).
New approach: ReapplyConfigForTarget() writes integration config
synchronously after config generation but before container restart,
with a no-op RestartStack since the caller handles restart.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Protected stacks like FileBrowser have no app.yaml so Deployed=false,
which caused the URL link condition to fail. Now also shows the URL
when the stack is protected.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
User feedback: these settings belong on the Beállítások (settings) page,
not the app description/details page. Moves both sections from app_info.html
to deploy.html and rewires data in deployHandler instead of appDetailHandler.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Containers with Docker healthchecks show 'starting' state for
~30s after compose up. The container is connectable, just hasn't
passed its healthcheck yet. Accept both running and starting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The goroutine fires immediately but needs the stack manager's
state to reflect 'running' before checking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Generic integration system for connecting deployed apps via toggle UI.
First handlers: OnlyOffice→FileBrowser (config.yaml patch) and
OnlyOffice→Nextcloud (occ CLI). Lifecycle hooks auto-suspend on
stop and re-apply on start.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Clear HealthProbe on StartStack/RestartStack so stale unhealthy state
isn't re-applied by RefreshStatus
- Use 10s probe interval for unhealthy/new stacks (nil HealthProbe probes
immediately on next tick), switch to normal 5m interval once healthy
- Scheduler frequency 1m → 10s to support fast probing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Always update CHANGELOG.md whenever you modified the code, and pushed to git!!
- IF controller feature changed (new/modify/remove) always update the relevant part of controller/README.md with the architectural change!!
## What this repo is
The **in-guest controller** — one per customer LXC, Docker-only, **holds NO Proxmox credentials**. It
owns the app domain: stack/deploy management, the Hungarian web UI, app-data backup, metrics,
integrations, git-sync, notifications. Disk/host/Proxmox concerns are delegated to the host agent via
`internal/agentapi`. Whole-guest backup (PBS vzdump) is the agent's, not ours.
## Project overview
**Don't confuse the two ex-"controllers":**`felhom-agent` (host, operator-tier, was
`proxmox-controller`) vs this repo (in-guest, was `deploy-felhom-compose`).
Creating a business (Felhom) for home-server deployment for Hungarian customers. This repository
(`deploy-felhom-compose`) contains the felhom-controller — a Go application that manages Docker
Compose stacks on customer hardware via a Hungarian-language web dashboard.
## Doing X → read Y
See `controller/README.md` for full architecture and status (update after each session, keep track of how different functions/features operate, like backup, monitoring, storage handling, app management, user settings, update workflow, notification system, etc-etc...).
See `CHANGELOG.md` for recent work (update after each session — see "Working with CHANGELOG.md" below).
See `CONTEXT.md` for current project state, decisions and roadmap (update after each session).
See `TASK.md` for the current task to implement (if it exists).
| Doing | Read |
|---|---|
| writing any new code | `REUSE.md` — canonical helpers, patterns, traps, seams |
| needing current state / roadmap | `CONTEXT.md` |
| needing a feature or architecture reference | `controller/README.md` |
| build, deploy, publish, verify a version | the **`felhom-build-deploy`** skill |
| writing or reviewing a test, fixing a bug | the **`felhom-testing`** skill |
- "Sablonok frissítése" button on Alkalmazások page
- Sync status exposed in `/api/system/info` response
## Debug logging
The controller has two-tier logging controlled by `logging.level` in `controller.yaml` (or `FELHOM_LOGGING_LEVEL` env var):
- **`info`** (default): Operation success/failure with elapsed time, post-start container states, scan counts
- **`debug`**: All of above plus env var keys per compose command, local image availability checks, compose command completion times, log fetch byte counts
Key patterns used in `internal/stacks/`:
-`time.Since(start)` for operation timing — always logged at INFO level
-`m.isDebug()` gates verbose output (env var keys, image checks)
-`truncateStr(s, 500)` caps stdout/stderr in error logs
-`logPostStartStatus()` runs async (goroutine + 3s sleep) after start/restart/update/deploy — never blocks or fails the operation
-`checkLocalImages()` parses compose YAML for `image:` lines, runs `docker image inspect` per image
- Env var **keys** are logged, never values (secrets safety)
## Important lessons learned
1.`PAPERLESS_OCR_LANGUAGES` (plural, with S) **installs** tesseract packs; `PAPERLESS_OCR_LANGUAGE` (singular) **selects** which to use
2.`docker compose restart` does NOT pick up new images — always use `docker compose up -d`
3. Go map iteration order is random — always sort before displaying in UI
4. Docker's `.State` field says "running" even for unhealthy containers — must parse `.Status` for health info
5. In-memory `Deployed` flag must be set BEFORE `docker compose up -d` (not after) — compose can take 30-60s for image pulls; revert both in-memory and disk on failure
6.`docker compose up -d` returns exit 0 even when containers crash-loop — post-start status check is essential for detecting failures
7. Mealie image has no wget/curl — use Python TCP socket check for healthcheck; set `start_period: 60s` for DB migration time
8. Always verify container images have the healthcheck tool (`wget`, `curl`, etc.) before using it — Alpine has BusyBox wget, Python images have `python3`
Guest 9201 is **bootstrap-managed — there is no compose file**;
`felhom-controller-bootstrap.service` runs the tag written in `/etc/felhom-controller-image`. Catalog
changes (`app-catalog-felhom.eu`) are picked up by controller sync ≤15 min, or via the "Sablonok
frissítése" button.
## Working with CHANGELOG.md
**DO NOT read the full file** — it is large (29K+ tokens) and will waste context or fail.
**DO NOT read the full file** — it is large and will waste context.
-**At session start:** Do NOT read CHANGELOG.md. Use`CONTEXT.md`and`controller/README.md` for current state.
-**To add a new entry:** Read only the top ~30 lines (`limit: 30`) to see the format and insertion point, then use Edit to insert the new entry after line 1 (`## Changelog`).
-**To check history:** Use Grep to search for specific topics instead of reading the file.
-Session start: `CONTEXT.md`+`controller/README.md` for current state.
-Adding an entry: Read only the top ~30 lines for format, then Edit-insert after line 1.
-History: Grep for topics instead of reading.
## End-of-session checklist
Before ending a session, always:
1.**Commit and push** all code changes (explicit paths; no `git add -A`).
2.**Build, push, and deploy** the new controller image, if controller code changed.
3.**`CHANGELOG.md`** — always, whenever code changed and was pushed.
4.**`CONTEXT.md`** — decisions made, state, what is next.
5.**`controller/README.md`** — whenever a feature was added, modified or removed.
6.**`REPORT.md`** — overwrite with this run's summary only.
7.**`REUSE.md`** — if a shared helper or pattern was added/changed/deprecated (same commit).
8.**Verify** the deployment (`docker ps` + logs).
1.**Commit and push** all code changes
2.**Build, push, and deploy** the new controller image (if controller code changed)
3.**Update CHANGELOG.md** with what was done
4.**Update CONTEXT.md** with decisions made, update architectural state and what's next
5.**Update controller/README.md** if architecture or features changed
6. **Verify** the deployment is working (check `docker ps` and logs)
<!--
WHY THIS FILE IS SHORT (2026-08-06, instruction-trim task).
Removed from here and rehomed, not lost:
- the `## Layout (verified against the tree)` block -> derivable by `ls internal/`; REUSE.md
carries the per-package seams and traps that the annotations were really for.
- the `!!! IMPORTANT !!!` header -> its two requirements are checklist items 3 and 5. One voice,
one place; a rule stated twice in one file is a rule that gets edited in one of them.
- the host/access table -> documentation/operations/nodes.md is the single home. The copy here
had drifted: it gave demo-felhom as plain root@192.168.0.162 (the LAN fallback, not the route),
pinned "agent 0.93.0" against the project's own no-versions-in-docs rule, and claimed no drill
VM was provisioned on demo-hp. Measured 2026-08-06: `qm list` on demo-hp shows VM 300
`drill-r50` present. felhom-agent/CLAUDE.md was right; this file was wrong.
- the "felhom-pve is back on the home LAN" block -> it was bookkeeping about a retired block; the
record is in documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md.
- the "Legacy: Windows workstation" block -> the workspace-root CLAUDE.md carries the full version.
- the gates/logging/coupling/UI paragraphs -> .claude/rules/*.md, which load when a matching file
is read instead of in every session.
Full per-block accounting: felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md
# DIAGNOSTIC — base-infra bring-up: why fresh guests are health=FAIL
**Scope:** read-only diagnosis. No code changes, no deploys, no state changes on guest 9201 or any guest.
**Subject:** demo guest **9201** (LXC), controller **v0.40.0**, online on the hub, Health = **FAIL: protected containers not running — traefik, cloudflared, filebrowser**.
**Date:** 2026-06-11. Evidence is live repo `file:line` + live 9201 output (secrets redacted).
---
## TL;DR — the one-line cause
**Nothing deploys the base/protected stack on a Proxmox bootstrap.** The traefik / cloudflared / filebrowser stacks were *only ever* created by the **bare-metal `scripts/docker-setup.sh`** (heredoc-generated compose files). The Proxmox golden→bootstrap path never runs that script, the controller has **no first-boot/reconcile/self-heal deploy** for the protected stacks, and the health loop only **detects** them missing. So on a provisioned guest there is no `/opt/docker/stacks`, no `traefik-public` network, and no infra containers — only `felhom-controller` itself runs, and health is permanently FAIL.
**Live confirmation (9201):**
```
pct status 9201 → running
docker ps → felhom-controller … Up (healthy) [ONLY container]
docker network ls → bridge, host, none [NO traefik-public]
ls /opt/docker/ → No such file or directory [no stacks dir at all]
docker logs felhom-controller | grep health
→ [monitor] Health check: status=fail (every cycle)
## A. Who is *supposed* to deploy the base stack, and why it never fires
**A1 — Every compose-up / DeployStack caller.**`DeployStack` has exactly **one** caller:
- [controller/internal/api/router.go:350](controller/internal/api/router.go#L350) — `r.stackMgr.DeployStack(deployReq)`, driven by the UI deploy form (`POST /api/stacks/{name}/deploy`).
A repo-wide grep for `EnsureBaseStack` / `deployProtected` / `BaseStack` / first-boot reconcile returns **nothing**. There is no programmatic deploy of the protected stacks anywhere. `runComposeDeploy` ([stacks/deploy.go:337](controller/internal/stacks/deploy.go#L337)) is reached only through `DeployStack`. The `deploy.go:24-25` "base" marking referenced in the brief is about *backup ordering* (filebrowser/traefik are flagged so backup skips them), not about deploying them.
**A2 — The Proxmox startup sequence, with the gap.** [controller/cmd/controller/main.go:56-711](controller/cmd/controller/main.go#L56):
```
71 config.LoadPermissive
93 bootstrap.MaybeIngest(...) ← comes up CONFIGURED (pulls yaml from hub, merges local_api)
After `MaybeIngest` the controller is fully configured (has CF tokens, domain, protected list) **but never deploys the protected stacks**. There is no `Ensure…`/`deploy` step between "configured" and "serving". This is the missing trigger.
**A3 — The health loop only reports, never self-heals.** [controller/internal/monitor/healthcheck.go:159-162](controller/internal/monitor/healthcheck.go#L159):
report.Issues=append(report.Issues,fmt.Sprintf("Protected container not running: %s",name))
}
```
→ `status="fail"` (healthcheck.go:177-181). The scheduler runs this every 5 min ([main.go:259](controller/cmd/controller/main.go#L259)) and pushes the FAIL to the hub. **No code path attempts to start or deploy the missing containers.** That is exactly the FAIL 9201 shows.
---
## B. Where the base-stack compose + config come from (decides what to bake)
**B4 — On 9201: nothing.**`/opt/docker/` does not exist (so no `/opt/docker/stacks`, no `traefik`, no `cloudflared`, no `filebrowser` dirs). The controller runs from a Docker **named volume** (`felhom-controller-data`), config at `/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml`. The 52 "available" stacks are the **catalog cache** (git-synced app templates), none deployed.
**B5 — Provenance of each infra app's compose + static config:** generated by **`scripts/docker-setup.sh`** heredocs — the **bare-metal** installer. They are **not** in the controller image, **not** in the app-catalog (`templates/` there = the 52 user apps), and **not** hub/asset-served.
Note the health check matches **container names** (`traefik`/`cloudflared`/`filebrowser`), so traefik+cloudflared living outside `stacks/` is fine for detection.
**Partial exception — filebrowser only.** The controller *can* regenerate filebrowser's compose+config: `generateFileBrowserCompose` ([web/handlers.go:1383](controller/internal/web/handlers.go#L1383)) + `generateFileBrowserConfig`, driven by `syncFileBrowserMounts` ([web/handlers.go:1295](controller/internal/web/handlers.go#L1295)). **But it refuses to create it the first time** — it early-returns if the compose is absent:
```go
// web/handlers.go:1304
if_,err:=os.Stat(composePath);os.IsNotExist(err){
s.logger.Printf("[WARN] ... FileBrowser stack not found at %s — skipping mount sync");return
}
```
There is **no traefik or cloudflared generator in the controller at all** (grep: zero hits for `traefik.yml`, cloudflared compose, or any `EnsureInfra`/`deployInfra`).
**B6 — Offline-capable?** On 9201 `assets.source_url: https://felhom.eu`, **`assets.sync_enabled: false`** — and assets are UI assets (logos), *not* infra compose. The controller needs **no hub fetch to deploy compose***in principle* (config is already local post-`MaybeIngest`), but today there is simply **no template to deploy** on a provisioned guest. First-boot deploy becomes offline-possible only once the templates are baked/embedded **and** a generator exists.
---
## C. Image provenance + bake feasibility (answers Viktor's question)
**C7 — Image refs (from the docker-setup.sh heredocs):**
| filebrowser | `gtstef/filebrowser:latest` (:1302, and controller's generator handlers.go:1397) | ❌ `:latest` | Docker Hub (public — note: `gtstef/`, **not** the official `filebrowser/filebrowser`) |
**C8 — Registry pull at first boot?** All three are **public Docker Hub** pulls → **no gitea private-registry credential needed** in the guest (good — none must ever be there). Without baking, first boot needs outbound Docker Hub access.
**C9 — Can `build-golden.sh` bake them?****Yes — same mechanism.** It already bakes the controller image with a plain pull-into-the-golden's-Docker: [felhom-agent/configs/build-golden.sh:71](../felhom-agent/configs/build-golden.sh#L71) `docker pull "$CONTROLLER_IMAGE"`, then logs out/removes the cred (:72) so nothing is baked but the image. Adding three more `docker pull` lines for traefik/cloudflared/filebrowser bakes them identically — and these are **public**, so they don't even need the build-time `docker login` the controller image uses.
- **Blocker / must-fix:** pin the two `:latest` tags before baking. A baked `:latest` drifts (the baked digest ≠ whatever `:latest` later resolves to), and any first-boot fallback pull would re-resolve `:latest` non-reproducibly. Pin to digests or explicit versions.
- **Baking the compose templates:** feasible but **not free** — it requires porting docker-setup.sh's traefik/cloudflared heredoc generators (static `traefik.yml`, ACME/cert-resolver block, dynamic config, the cloudflared compose) into the controller as Go templates rendered from `controller.yaml`. The controller today has only the **filebrowser** generator. This is the real work item; the image bake is trivial by comparison.
**C10 — Running-container bake (the hard line):****No infra app is safe to bake as a *running* container.** Each is per-customer-parameterized with secrets injected at run:
- cloudflared run env `TUNNEL_TOKEN=${CF_TUNNEL_TOKEN}` ([docker-setup.sh:1099](scripts/docker-setup.sh#L1099)) — per-customer tunnel token → **must NOT** be baked running.
- traefik consumes the per-customer CF API token + ACME email + domain (see D) → **must NOT** be baked running.
Every customer parameter the base stack needs is **already in the local config** after `MaybeIngest`. Nothing additional must be fetched to render them.
---
## E. Hostname / CT-name (diagnose now, fix later)
**E12 — Reported hostname is the Docker container ID.** [controller/internal/report/builder.go:75](controller/internal/report/builder.go#L75) `Hostname: staticInfo.Hostname` ← `os.Hostname()`. The controller runs inside Docker, and the golden bootstrap `docker run` sets **no `--hostname`** ([felhom-agent/configs/build-golden.sh:94](../felhom-agent/configs/build-golden.sh#L94)) → `os.Hostname()` returns the container ID.
- Live: `docker inspect felhom-controller --format '{{.Config.Hostname}}'` → **`3dff0fe73b5c`** (the value reported to the hub).
- **Insertion point:** the bootstrap unit's `docker run` (build-golden.sh:94). It already reads `/etc/felhom-bootstrap/bootstrap.json`; add `--hostname <customer-id>` parsed from that file. The id is present — `bootstrap.json` carries `customer.id` (the pull target), per [controller/internal/bootstrap/bootstrap.go:66-68](controller/internal/bootstrap/bootstrap.go#L66) (`BootstrapCustomer.ID`). Feasible with a small `grep`/`jq` in the baked `felhom-controller-bootstrap.sh` heredoc.
**E13 — Proxmox CT/LXC hostname is `felhom-golden`.** The golden is created `--hostname felhom-golden` ([build-golden.sh:38](../felhom-agent/configs/build-golden.sh#L38)); `/etc/hostname` is removed at minimize (:146) but the **PVE container-config hostname is not reset on restore**, so the guest inherits `felhom-golden`.
- **The mechanism to fix it already exists in the agent:** [felhom-agent/internal/reconcile/bringup.go:303-304](../felhom-agent/internal/reconcile/bringup.go#L303) sets `params["hostname"] = spec.Hostname` (via `SetConfig` / `pct set`) when `Mode==ModeProvision && Hostname!=""`. The provision path passes `Hostname: a.hostname` ([felhom-agent/cmd/felhom-agent/main.go:1041](../felhom-agent/cmd/felhom-agent/main.go#L1041)) from a `-hostname` flag.
- **Why 9201 still shows `felhom-golden`:** it was provisioned **without** a `-hostname` value → `spec.Hostname==""` → the `SetConfig` hostname step is skipped → the golden's name persists. **Fix = wire the provision back-half to pass `Hostname=<customer-id>` (sanitized) into `BringUpSpec`.** No new mechanism needed.
> These are two **independent** layers: E13 fixes the Proxmox CT name + LXC hostname; E12 fixes what the *controller* reports to the hub (the Docker container's `os.Hostname()`). Fixing only one leaves the other wrong.
---
## F. Recommended insertion point for first-boot base-stack bring-up
**Recommendation: option (a) — the controller deploys its own base stack on first configured boot, and self-heals it when missing.**
Place an `EnsureBaseInfra()` step in [cmd/controller/main.go](controller/cmd/controller/main.go) **after** `stackMgr.ScanStacks()` (line ~144) and Docker is confirmed reachable, and additionally invoke it from the 5-min `system-health` job when `checkProtectedContainers` reports any protected container missing (turn healthcheck.go's detection into a reconcile trigger).
**Why (a):**
- The full config (CF tunnel token, CF API token, domain, email, storage paths) is **already local** after `MaybeIngest` (Section D) — no secret needs to enter the golden.
- The controller already **owns stack deployment** (`stacks.Manager`, `docker compose` via the mounted socket) and already has the **filebrowser generator** — extend the same pattern to traefik/cloudflared.
- The health loop already **detects** the missing protected set; making it reconcile is the natural, idempotent, self-healing design (survives a wiped/half-deployed guest).
- Keeps customer secrets out of the golden and out of the agent's bootstrap payload.
**Why not the others:**
- (b) golden bootstrap-unit step → would have to render per-customer traefik/cloudflared config in shell and risks putting/handling secrets in the unit; duplicates logic the controller is better placed to own.
- (c) headless reuse of the setup wizard's deploy path → the wizard **never deployed** the base stack either (it only writes `controller.yaml`, [setup/handlers.go:398-514](controller/internal/setup/handlers.go#L398)); there is no deploy path to reuse.
**Prerequisites / ordering constraints for (a):**
1. **Port the traefik + cloudflared compose/config generators into the controller** (Go templates from `controller.yaml`). This is the main build item; filebrowser's generator already exists but must **drop its "skip if absent" early-return** ([web/handlers.go:1304](controller/internal/web/handlers.go#L1304)) so it can create on first boot.
2. **Bake the three infra images (pinned) into the golden** (build-golden.sh) so first-boot deploy is offline-capable; pin the two `:latest` tags.
3. **Create the `traefik-public` docker network** + the stack dirs as part of bring-up (absent on 9201 today).
4. Run only when configured (post-`MaybeIngest`, `NeedsSetup==false`) and after Docker is reachable; make it idempotent (no-op when the protected containers are already up).
---
## G. Additional gap surfaced (flag — needs validation before the spec)
**The bootstrap `docker run` does not bind-mount the stacks dir or `/opt/docker` from the LXC host.** It mounts only ([build-golden.sh:94-99](../felhom-agent/configs/build-golden.sh#L94)):
So `paths.stacks_dir = /opt/docker/stacks` exists **only inside the controller container**, while `docker compose up` (invoked by the controller over the shared socket) is executed by the **host LXC's** Docker daemon. Compose files are read by the in-container CLI, but **bind-mount sources** in those compose files (e.g. traefik's `./traefik.yml:/etc/traefik/...`, filebrowser's `./config.yaml`, app `HDD_PATH` mounts) are resolved by the **daemon on the host filesystem**, where `/opt/docker/stacks/...` does **not** exist. On bare metal this worked because `/opt/docker/stacks` was a shared host bind-mount into the controller.
This is a **path-namespace mismatch that affects ALL stack deploys** (every catalog app, not just base infra), so it sits squarely in the blast radius of "stand up the base stack." It is inferred from the mount topology + how the controller shells `docker compose` with `cmd.Dir=stackDir`; it was **not** live-exercised here (no deploy attempted, per the read-only rule). **Recommend the bring-up spec validate this explicitly** and, if confirmed, add a host bind-mount (e.g. `-v /opt/docker/stacks:/opt/docker/stacks`) to the bootstrap `docker run` so container and daemon agree on the path.
---
## Evidence index (live repo file:line)
- No base-stack deploy caller: [api/router.go:350](controller/internal/api/router.go#L350) is the sole `DeployStack` caller; startup [cmd/controller/main.go:56-711](controller/cmd/controller/main.go#L56).
`felhom.eu` (documentation only — no hub code, no manifest bump, no ArgoCD sync)
---
## 1. Confirmed baselines used (as read at the start of the run)
| Repo | `main` @ start | Version | → Shipped |
|------|----------------|---------|-----------|
| felhom-controller | `3e3ee94b7bbe6b66663c468e22aa86616365a45a` | v0.214.0 | **v0.215.0**, then **v0.216.0** (a defect found live in v0.215.0 — §14) |
**That pair is the proof.** The identical event, differing only in one word of the severity string,
is the difference between *delivered to the operator* and *stored as an informational notice and
delivered to nobody*. This is the first time this leg has been observed end to end.
Only the operator leg fired because **demo-hp has no `customer_notifications` row at all** (no
customer email, no `enabled_events`), so no customer row was possible for either push — verified
directly, not assumed. **One real email was sent to the operator**, as the task anticipated.
---
## 9. NOT yet live-validated — stated explicitly
**The Fail-from-counters path has never fired on real hardware.** Everything in §4/§5 exercises it
against the committed fixture's values in unit tests only. The live legs above prove the *negative*
(no false alert on three healthy disks) and the *severity wire* (end to end, through the hub) — they
do **not** prove a live disk reaching Hiba. The fixture tests must not be read as a live proof.
Tracked as **R-332 (WATCHING)**. Closing condition: a live disk reaching Hiba from counters, or a
deliberate injection through the real pipeline (agent `/disks` → controller check → hub event) — not
a hand-set verdict.
**One item originally listed here has since been proven live** and is no longer part of this gap: the
**persisted state surviving a controller restart**. The v0.215.0 → v0.216.0 redeploy destroyed and
rebuilt the container, and the new one read back a `changed_at` written by the previous version rather
than re-baselining — see §14. What remains unproven is the stronger half: an already-**alerted** disk
not re-alerting after a restart, which needs a disk that has actually alerted. The drive that produced the fixture lives in DooPlex, which is Tier 2 and never a
drill target; the demo boxes are all-flash and healthy.
---
## 10. Teardown
**This run provisioned nothing** — no VM, no guest, no hub customer record, no storage. Nothing was
formatted, mounted, unmounted, repaired or written on any monitored disk; the only write is the
controller's own `disk-health-state.json` inside its data volume.
Disposition of what the run did create:
- **Two synthetic hub events (`events` id 2964, 2965) and one `notification_log` row (id 689)** on the
live hub. **Left in place deliberately.** Both messages are self-labelling
(`"R-328 severity probe (…) - synthetic, no real disk fault"`), and deleting rows from the
production hub DB is a riskier act than leaving two clearly-marked probe rows. Named here so they
are not mistaken later for a real disk fault on demo-hp.
- **One real operator email** resulting from row 689.
- A local copy of `hub.db`/`-wal`/`-shm` in the session scratchpad only (not committed, not exported).
---
## 11. Register rows
| Row | State | Owner |
|-----|-------|-------|
| **R-328** — the severity drop: `"warn"` coerced to `info`, emailed to nobody | **CLOSED** (controller v0.215.0), proven live side by side | CC |
| **R-329** — `app_start_failed` carries the identical defect | READY — **not fixed here**; needs a decision on whether it should notify at all | Viktor |
| **R-330** — Phase 2: collect SMART attrs 187/199/188 + persist samples | READY — a declared wire change, hub models it in the same session under G-1 | CC |
| **R-331** — Phase 3: growth-rate detection; revisit the static 64 | READY, blocked on R-330 | CC |
| **R-332** — the Fail path has never fired on real hardware | **WATCHING** | CC |
| **R-333** — NVMe temperature bands; agent `smartctl` has no `-n standby` | READY (S each) | Viktor decides (a); CC does (b) |
| **R-334** — released with no golden carrying it (gate waiver) | READY — now applies to **v0.216.0** | CC bakes; **Viktor vouches** |
| **R-335** — one physical disk walked twice per run, sustaining against itself | **CLOSED** (controller v0.216.0) | CC |
`smartd`-on-DooPlex-alerts-nobody is recorded in `DIAG-smart-passed-trap-2026-08-14.md` §8 as the
same shape one layer out.
---
## 12. Observations — noticed, NOT acted on
1.**`app_start_failed` has the identical severity defect** (`notifier.go` ~L546, `"warn"`). Left
untouched per scope. It needs a prior decision — should a stopped app email the customer at all? —
because flipping the string alone converts a silent event into a mail flood on a crash-looping box.
**R-329.**
2.**The 55/60 °C bands are spinning-disk bands being applied to NVMe, and this is close to biting.**
Adopted unchanged from the operator's Prometheus config by explicit decision — but demo-hp's
**healthy** Toshiba NVMe idles at **53 °C**, i.e. **2 °C below Figyelmeztetés and 7 °C below Hiba**,
and NVMe routinely passes 60 °C under sustained write with no fault. As shipped, a healthy customer
NVMe under load can be reported as **Hiba** — the single worst outcome this feature can produce, and
the one leg 1 exists to guard. Not changed here because the threshold is a stated, settled operator
decision; flagged rather than overridden. **R-333(a) — recommend splitting the bands by device
class, or dropping them for NVMe and relying on `critical_warning`.**
3.**The agent runs bare `smartctl -a -j` with no `-n standby`**
(`felhom-agent/internal/storage/hostops.go:368`), so every poll wakes a spun-down drive, and 6h → 1h
multiplies that by six. Recorded, not acted on, per the task's instruction. demo-hp is all-flash so
the measurement could not reveal it. Mitigating datum from the fixture: the failing drive logged
only **3375 load cycles in 60505 power-on hours** (~one per 18 h), so this duty cycle barely spins
down at all. **R-333(b).**
4.**`source ~/.config/credentials` prints two recovery codes to the terminal.** The file contains
hyphenated keys (`R_DEMO-FELHOM`, `R_DEMO-HP`) that bash cannot assign, so sourcing it emits
`command not found` errors **containing the secret values**. Anything that sources that file leaks
them into logs, scrollback and transcripts. Not a code defect and out of scope; worth quoting
values from it by other means, or renaming the keys.
5.**`golden_currency_gate.py` has no waiver parser.** Its own failure text says *"record a waiver in
`OPEN-ITEMS.md` — never a bypass"*, but nothing reads such a waiver, so the only way past it is the
bypass it warns against. See §13.
---
## 13. Deviations, stated plainly
- **`git push --no-verify` was used once**, on the `felhom.eu` docs push (`767960b`), and only there.
Cause: `golden_currency_gate.py` correctly convicts the fact that controller **v0.215.0 is released
and no golden carries it** (newest bake 0.214.0), so a *newly installed* machine would receive
0.214.0 — without the severity fix. A golden bake was out of the task's scope, and its second half
(vouching in the hub's day-0 artifact manifest) is operator-password-gated, so CC cannot complete it;
a baked-but-unvouched golden is worse than none. Recorded as **R-334** with the bake+vouch owners
named. CI re-runs the same entry point and will mail the operator. The running fleet is unaffected.
- **One pre-existing test changed meaning by design:** `TestDiskVerdictFor`'s
`critical_warning>0 → warn` case is now `→ fail` (truth-table row 4 — NVMe's own critical flag is a
device declaration, not a drifting counter). `TestDiskHealthCheck_DegradationOnce` and its siblings
were rewritten into the scenario groups because they encoded the pre-v0.215.0 single-alert behaviour
the task deliberately replaces (Scenario C).
---
## 14. R-335 — a defect in v0.215.0, found live, fixed as v0.216.0
**How it was found.** Not by a test and not by review: by reading the release's own **positive
observable** against the release's own **persisted artefact**. The hourly check logged *"3 disk(s)
evaluated"*; `disk-health-state.json` held **two** records. Two artefacts that should have agreed did
not.
**Cause.** demo-hp's `c11-scratch` and `felhom-backup` are the same physical NVMe (`/dev/nvme0n1`) and
resolve to the same `diskKey`, so one disk was walked twice in a single run.
**Why it mattered.**`RunDiskHealthCheck` writes a disk's new record before the next entry reads it, so
the **second** copy of an aliased disk consumed the **first** copy's write as its prior. The disk
therefore **sustained against itself and reached Hiba on a first sighting** — defeating truth-table
row 6, the single rule separating a one-hour benign excursion from a false critical alert — and would
have emitted **two identical events** for one drive.
**Severity in practice: latent, not active.** Nothing fired on demo-hp because all three entries are
healthy with zero counters. But any aliased disk developing one pending sector would have gone
straight to Hiba, which is precisely the outcome §8 leg 1 exists to prevent. Aliasing is not exotic —
it is the *normal* shape whenever a box has two PVE storage entries on one physical device.
**Fix (v0.216.0, `90f2545`).** Each `diskKey` is evaluated once per run. Both entries stay marked
`seen`, so neither is mistaken for a disappeared disk, and the card still renders **both** storage
rows — the dedup is about state and alerts, not display. Pinned by
`TestDiskCheck_SameDiskTwiceIsEvaluatedOnce`, red-proof run and reverted (§5).
**Deployed:**`gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy)`.
**Confirming cycle on v0.216.0 — CONFIRMED LIVE, 09:31:35Z:**
```
live image: gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up About an hour (healthy)
> Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid.
> Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper.
> Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.
## 1. Canonical helpers (MUST reuse — do not reinvent)
### Paths & namespaces (felhom-data layout)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `NamespaceRoot` | controller/internal/appbackup/paths.go | `(drivePath string, inGuestDrive bool) string` | Resolve felhom-data root for a drive | `inGuestDrive=true` returns path AS-IS (Model A: guest mount IS the ns root); false appends `felhom-data`. Never double-nest |
| `PrimaryBackupPath` / `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath` | controller/internal/appbackup/paths.go | `(nsRoot[, stackName]) string` | All backup dir layout | Take the NAMESPACE ROOT, not a bare drive path |
| `AppDBDumpPath` / `AppVolumeDumpPath` / `AppDataDir` | controller/internal/appbackup/paths.go | `(nsRoot, stackName) string` | Per-app dump/data dirs | Same nsRoot contract. `AppDataDir`'s final segment is the app's real appdata dir NAME — NOT always the stack name (paperless-ngx → `paperless`); resolve via `AppDataDirNames` first (F-S2/F-S3) |
| `AppDataDirNames` / `AppDataBindsPresent` | controller/internal/appbackup/paths.go | `(hddPath, stackName string, hddMounts []string) []string` / `(hddPath, hddMounts) bool` | Resolve the real `appdata/<name>` dir(s) from compose `${HDD_PATH}` binds (F-S2/F-S3) | `hddMounts` = ParseComposeHDDMounts shape. Deduped+sorted; falls back to `[stackName]` when no appdata bind. Tier-2 (`backup.Manager.tier2AppDataName`) refuses N>1; migrate (`stacks.Manager.ResolveAppDataDirNames`) loops N. `BindsPresent` drives the WARN-on-missing-declared-dir |
| `UserdataDir` / `ImportDir` / `EnsureUserdataSkeleton` / `EnsureDirOwned` | controller/internal/appbackup/userdata.go | `(nsRoot)` / `(nsRoot)` / `(nsRoot, dirs []string)` / `(path, gid int)` | userdata/ tree w/ 2775 setgid gid-1000 convention. **R-75:**`ImportDir` is the CANONICAL drop-zone (`<nsRoot>/userdata/import`) and callers MUST resolve it against the SYSTEM namespace, never an app's HDD_PATH — use `stacks.Manager.GetImportRoot()`. `EnsureUserdataSkeleton` now takes the dir set: build it with `BuildUserdataSkeleton(DeriveUserdataDirs(stacksDir))`, or via `Manager.EnsureUserdataSkeleton` / `web.Server.ensureUserdataSkeleton`. | Linux-only chown via build-tag twin userdata_linux.go. **The set MUST stay sorted** — `fbNeedsRecreate` force-recreates FileBrowser on any byte diff and the naive map-order derivation measured 20/20 distinct (SPIKE P6). `UserdataSkeletonCarry()` is the old hardcoded list, retained forever so derivation can only ADD (zero removals). |
| `BuildUserdataSkeleton` / `UserdataSkeletonCarry` / `DeriveUserdataDirs` | appbackup/userdata.go, stacks/skeleton_derive.go | `([]string)` / `()` / `(stacksDir)` | catalog-derived userdata skeleton (R-75) | Derives `${USERDATA_PATH}` binds only — `${IMPORT_PATH}` is NOT part of a drive skeleton (one root, system drive, `Manager.EnsureImportRoot`). Do NOT wire the catalog sync to `SyncFileBrowserMounts`. |
| `appbackup.ValidateRelPath` / `ValidRoot` | controller/internal/appbackup/classify.go | `(root, path)` / `(root)` | THE single path-safety refusal set for every `${VAR}`-relative catalog path | Shared by `backup:` and `data_paths:`. **Do not write a second path validator.** |
| `web.fileBrowserLink` / `importFolderLink` | controller/internal/web/filebrowser_link.go | `(domain, sourceName, relPath)` | FileBrowser Quantum deep link | Template read out of the shipped router (SPIKE P2). **`url.PathEscape` per segment — NEVER `QueryEscape`** (space→`+` is a literal plus in a path). Let `html/template` do the attribute escaping; do not pre-escape. |
| `stablePathForName` / `agentWhere` | controller/internal/web/intermediary.go | `(name/registeredPath) string` | Map registry stable path `/mnt/felhom-drives/<n>` ↔ raw agent mount | Registry stores STABLE path; agent ops take the RAW mount — always convert |
| `offsiteRestoreRootFor` | controller/internal/backup/offbox_verify_copies.go | `(drivePath string) string` | THE only place `backups/offsite-restore` is spelled | `offboxRestoreScratchDir` builds on it — the listing/delete surface MUST resolve byte-identical paths to what the restore wrote. Do not re-hardcode the segments (they were open-coded in 3 places before v0.147.0) |
| `ProtectedHDDPaths` | controller/internal/stacks/delete.go | `(hddPath string) map[string]bool` | Never-delete set (root, appdata, backups, media, legacy felhom-data) | Consult before ANY recursive delete under a drive |
### Subprocess + timeout + exit-code discipline
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `Manager.composeExec` / `composeExecCustomEnv` | controller/internal/stacks/manager.go | `(dir string, [env,] args...) (string, error)` | ALL docker-compose invocations | Logs env KEYS only (secrets safety), truncates output to 500, extracts exit code; `up` triggers userdata pre-create belt. NO timeout — see §3 |
| `runCommand` / `runCommandStdin` | controller/internal/selfupdate/updater.go | `(name, args...) (string, error)` | docker CLI in updater | stdin variant for `docker login --password-stdin` (no secret in argv); package VARS since v0.112.0 — override in tests (fakeRunner in registry_anon_test.go) |
| `parseWWWAuthenticate` + `fetchAnonymousToken` | controller/internal/selfupdate/updater.go | Bearer-challenge parse + anonymous Docker v2 token | Any credential-free registry API access | realm comes FROM THE HEADER (never hardcode a token URL); denial = errAnonymousDenied, never "credentials missing" |
| `offboxRedirectTo` | controller/internal/web/offbox_handlers.go | `(w, r, page, msg string, isErr bool)` | Same, to an EXPLICIT page | **TRAP (fixed v0.154.0): the separator is chosen, not `"?"`.** Targets may already carry a query — the R-48 wizard is `/backups/restore/app?name=<app>` — and a hardcoded `"?"` buries the flash inside the previous parameter's value |
| `restoreOpInFlight` + `hasRecentRestoreResult` | controller/internal/web/restore_wizard.go | `(backup.RestoreOpStatus) bool` / `(st, app, now) bool` | THE "is a restore running / did one just finish" display reads | **TRAP (v0.154.0 shipped this bug): `Manager` has TWO running flags.**`IsRunning()` reads the CONCURRENCY flag, acquired inside the goroutine — and `RestoreOffboxScratch` never acquires it, so it is false for the whole verification restore. Display must read `RestoreStatus().Running` (set synchronously by `BeginRestoreOp`). Read the status ONCE per render or the strip and the suppression can disagree. `hasRecentRestoreResult` is app-bound and window-bounded — a process-wide result must not light another app's „Eredmény" |
| `restoreWizardPath` / `deriveWizardStep` / `resolveWizardApp` | controller/internal/web/restore_wizard.go | `(app) string` / `(restoreWizardInput) restoreWizardView` / `([]OffboxAppRow, name) *OffboxAppRow` | R-48 offsite restore wizard: URL builder + the PURE step/unlock derivation + the app-resolution refusals | The step is **never** taken from the request. Precedence is load-bearing: op-running outranks a stale `?full_prep=`, else a commit button reappears mid-restore. Truth table + red-proof: `restore_wizard_test.go`. Adding a form here that posts anywhere new breaks `TestRestoreWizard_NoNewMutationEndpoints`**by design** — R-48 adds no mutation surface |
| `validStackName` | controller/internal/web/validate.go | `(name string) bool` | Any stack name from a request | Single-segment, no `/ \ ..` — blocks path traversal into stacks/userdata |
| `ValidateSegment` | controller/internal/appexport/validate.go | `(kind, s string) error` | Any attacker-controlled path segment (.fab manifest fields) | CTRL-001 guard; deliberately NOT for dotfile ConfigFiles |
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `backup.ErrOffboxSealedPackageHeld` + `IsOffboxSealedPackageHeld` + `sealedPackageHeld` + `OffboxAwaitingRecoveryKey` (R-241, v0.206.0) | controller/internal/backup/offbox.go | sentinel; `(error) bool`; `() bool`; `() bool` | **THE MINT GUARD** — a box never creates a repository key while the hub holds a sealed package for it | **The guard is a CONJUNCTION** (package held AND no key present). Widening it to "never mint" leaves a first-time box unable to start, waiting for a package that will never exist — pinned by `TestR241_ScenarioB_FirstTimeBoxStillMints`. **The refusal is a HOLDING state, not a failure:**`ApplyOffsiteTarget` catches the sentinel and still writes the transport, so `/recovery`'s synchronous tier-up (R-219) can bring the tier up the instant the key arrives; returning the error instead leaves `needsOffsiteCredential` true and the hub re-staging a consumed credential for ever. `OffboxAwaitingRecoveryKey` is **DERIVED, never stored** — and **`t.Enabled` is load-bearing in it**: a customer who switched off-site OFF is not awaiting anything (the Scenario-E carve-out `needsOffsiteCredential` makes two functions above; the first draft omitted it and an existing test caught it). A nil settings store reads as "no package held" — a transient read failure must never become a permanently-held tier |
| `settings.HubEscrowKeySHA256` + `SetHubEscrowKeySHA256` / `GetHubEscrowKeySHA256`, and `OffsiteRecoveryOffer`**shape (c)** (R-241, v0.206.0) | controller/internal/settings/settings.go, controller/internal/backup/offbox.go | `(sha, checkedAt string) error` / `() (string, string)` | **THE DISCRIMINATOR the recovery screen asks** — does the hub hold a package for a key other than the one we use? | **The comparison was ALREADY computed on every ACK since SLICE 3 and persisted nowhere** — that is R-241's second half. Wire the recorder in `main.go`'s `EscrowAutoConfirmer` literal or shape (c) reads an empty hash for ever and the fix ships INERT (pinned by `TestMainWiresRecordEscrowKeyHash`). **§7.2 staleness, decided:** a KNOWN DIFFERENCE offers **however old the reading** — age is deliberately NOT gated on, because gating makes a box offline from the hub silently stop offering; an **ABSENT hash falls back to (a)/(b)** and does NOT offer, because `""` is the hub positively saying its package seals no key (legacy hash-less escrow), not an unknown. `CheckedAt` is for diagnosis, never a gate |
| `backup.AbandonStatus` / `AbandonSweep` / `CancelAbandon` / `ClearAbandonPurgeIfConfirmed` / `ExtendAbandon` / `StopAbandon` + `AbandonGraceDays` (R-241, v0.206.0) | controller/internal/backup/offbox_abandon.go | see file | **The 14-day abandonment countdown** — the ONLY thing in the product that deletes a customer's off-site history | **BOTH HALVES OR NEITHER.** The set-aside store and the sealed package that protects it are two halves of one thing; removing only one leaves a package that opens nothing, or ciphertext nobody can decrypt. Not atomic across two machines, so it is a **two-phase commit**: delete the store, set `AbandonPurgeRequested`, and keep declaring it until the hub's ACK stops reporting a superseded package — the confirmation rides the SAME ACK as the request. **The countdown starts in `ResetOrphanedRepo`, NOT in the shared `resetOrphanedRepo`** — the helper is also the UNCLAIMED auto-reset, where nobody decided anything. **The recovery offer stays reachable for the whole grace** (a grace in which recovery is impossible is decorative). **Drive it with `SetOffboxClock`, never a shortened live timer** (§7.4). A transport failure leaves the countdown DUE so tomorrow retries; the operator levers REFUSE rather than no-op when nothing is running or the store is already gone |
| `settings.SyncRecoveryOfferEpoch` / `PostponeRecoveryNoticeForEpoch` / `OptOutRecoveryRemindersForEpoch` + `web.recoveryBannerCookie` (R-241, v0.206.0) | controller/internal/settings/settings.go, controller/internal/web/recovery_handlers.go | `(offered bool, now) (RecoveryOfferView, error)` | **The offer EPOCH** — "once per entry into the offered state", not once ever | **Sync the epoch FIRST and UNCONDITIONALLY in `recoveryInterrupts`.** The first draft returned early when the offer was false, so the FALLING edge was never recorded, `RecoveryOfferActive` stayed true through a settled period, and the next entry counted as a continuation — **the exact defect the epoch exists to fix, reintroduced inside the fix**. Dismissals are recorded against the epoch they were made in, so a fresh entry resets them **by arithmetic**, with nothing to clear. **Three levers, three scopes, and NONE removes the entry point on `/backups/remote`:** the banner cookie is a browser SESSION cookie (no MaxAge — cleared on login) and persists nothing; the reminder opt-out is durable but silences the BANNER ONLY; "most nem" suppresses the full page only |
| `atomicWrite` | controller/internal/backup/recovery_unit.go | `(path, data, perm) error` | Atomic file writes (backup pkg) | tmp+rename; no dir creation, no fallback |
| `Settings.save` (unexported) | controller/internal/settings/settings.go | via mutator methods only | ALL settings.json persistence | tmp+rename, then `.bak` last-known-good AFTER rename succeeds. Never write settings.json by hand |
| `backup.SharesPseudoStack` / `DisplayStackName` | controller/internal/backup/shares_payload.go | `"_shares"` / `(key) string` | THE reserved key for the shares source (restic tag, `backups/secondary/_shares`, CrossDriveBackup record) + its display mapping | NEVER let the raw key reach a Hungarian surface — map at the notification/prose boundary ONLY; the persisted `EnlargedBlocked` set and the templates index by the RAW key |
| `Manager.buildSharesPayload` / `classifiedShares` | controller/internal/backup/shares_payload.go | `() (dir, passdbOK, error)` / `() []classifiedShare` | the definitions+credential payload and the availability-filtered share set both tiers read | payload is SECRET-BEARING (0600 passdb.tar) — never log its bytes/name at INFO. `classifiedShares` is the single place a dead mount is dropped, so both jobs agree |
| `Manager.selectTier2TargetFrom` | controller/internal/backup/tier2.go | `(stack, sourceDrive, fullSize, stateOnlySize) (*Tier2Target, error)` | tier-2 target choice with the source drive supplied EXPLICITLY | the seam the shares job reuses — NEVER fork the headroom math; `selectTier2Target` is now a thin wrapper over it |
| `Manager.tier2ReconcileRoots` | controller/internal/backup/tier2.go | `(destBase, roots, legRels)` | staleness pruning with explicit dest roots | pure extraction from `tier2Reconcile` (which now calls it with `hdd`/`userdata`); reuse it rather than writing a second pruner |
| `Manager.liveShareRootOK` / `scratchJoin` | controller/internal/backup/shares_restore.go | `(dst) bool` / `(scratch, abs) string` | THE place guard for shares restore + scratch path reconstruction | a snapshot is UNTRUSTED layout input: require a STRICT descendant of a live registered root, refuse `..` and the drive root itself. `scratchJoin` strips the volume name — plain `filepath.Join` splices a drive letter mid-path |
| `infra.SambaContainerName` / `SambaPassdbVolume` / `SambaPassdbMount` | controller/internal/infra/samba.go | consts | single source of truth for the samba container identity | the compose renderer interpolates them; stacks/backup/monitor read them. The CONTAINER name (`felhom-samba`) is NOT the stack name (`samba`) — `EffectiveProtected` needs the container one |
| `sambaWriteAtomic` | controller/internal/stacks/samba.go | `(path, data, mode) error` | samba smb.conf/compose writes | tmp+**fsync**+rename (the only one of these that fsyncs). Fourth atomic-write helper in the tree — see §6 |
| `Loop.writeMarker` / `Recover` | controller/internal/quiesce/quiesce.go | `(m Marker)` / `()` | Quiesce crash-safety | Marker written BEFORE stopping stacks; Recover restarts stranded stacks at boot |
| `quiesce.TieredBackend` + `Loop.resolveDueTiers` / `quiesceAndPollTiers` | controller/internal/quiesce/tiers.go, quiesce.go | `Tiers/DueFor/StartBackupFor/BackupStatusFor`; `resolveDueTiers(ctx) ([]dueTier,bool,error)` | THE R-82 multi-tier backup schedule — several whole-guest tiers (local daily + PBS weekly) reconciled into ONE quiesce window | **Both tiers due ⇒ ONE stop/start pair**, never two (two = two app outages for one night). Tiers run SEQUENTIALLY (vzdump holds a guest lock) and the app stays down until the LAST tier snapshots — resuming earlier loses app-consistency on the DR tier. Order is fast-first (agent advertises primary first) or downtime blows up. `ErrTiersUnsupported` (route 404) ⇒ pre-R-82 agent ⇒ degrade to the untargeted path and **STILL BACK UP** — never read it as "nothing due". |
| `quiesce.failureBreaker` + `Loop.dropBackedOffTiers` / `noteTierFailure` / `noteTierSuccess` | controller/internal/quiesce/breaker.go, quiesce.go | `blocked/recordFailure/recordSuccess(target, now)`; `backoffFor(n) time.Duration` | **R-88** — a tier whose backups keep failing stops re-quiescing. Backoff 15m→30m→1h→2h→4h (cap), reset on success | **It gates the QUIESCE, not the backup** — the harm was never the failing backup, it was the app outage taken to attempt it, so backed-off tiers are dropped from the due set BEFORE any stack is stopped. **Per TARGET** — a broken offsite tier must never suppress a healthy local one (`TestBreaker_OneFailingTierDoesNotSuppressAHealthyOne`). **Never permanent** — the cap bounds the retry INTERVAL, it never stops retrying; a latched breaker is a silent backup outage, worse than the loop it replaces. **`TriggerNow` is never gated** (it already bypasses due-ness and the window gate), though a manual run still RECORDS its outcome. **`stillRunning` is NOT a failure** — a first full offsite snapshot legitimately runs for hours. State is **in-memory on purpose**: a restart forgets the backoff and re-attempts, which is the cheap direction to fail. Log the deferral ONCE when armed, never per tick. |
| `quiesce.TierNotifier` + `Loop.SetTierNotifier` / `noteTierFailure` / `noteTierSuccess` | controller/internal/quiesce/breaker.go, quiesce.go | `BackupFailed(tier,msg,err)` / `BackupRecovered(tier,msg)`; `SetTierNotifier(n)` INIT-ONLY | **R-97a** — the whole-guest backup tier reports its outcome to the hub | A **seam, not an import** — quiesce keeps no dependency on `internal/notify` (same reason `windowStartFn` is injected). Wired by a setter because main.go builds the notifier AFTER the loop; `nil` = unprovisioned guest, not an error. **Edge-triggered:** failure fires only when the breaker ARMS (`n == 1`), never per retry — the cadence is 15m/30m/1h/2h/4h and an event per attempt is an inbox nobody reads. Recovery rides `recordSuccess`'s existing bool. **Event types are OPERATOR-ONLY** (`whole_guest_backup_failed`/`_recovered`, hub >= v0.78.0) — NOT `backup_failed`, which has a customerMessages entry AND sits in live `enabled_events`, so it would email the CUSTOMER about a backup they cannot act on. `WholeGuestBackupDetails.Tier` is load-bearing: the hub keys its per-tier cooldown on it. |
| `quiesce.Loop.SuppressedStacks` + `markQuiesced` / `markUnquiesced` | controller/internal/quiesce/suppress.go | `() map[string]bool` (nil-safe on a nil *Loop) | **R-97b** — an app THIS controller stopped for a backup is not a fault | Consumed at the SINGLE derivation point `classifyRunStates` (which computes both the banner dead-list and the notifier Down-set — keep it one place). **Cycle-keyed, not state-based:** v0.164.0's `!= StateStopped` filter cannot see an app caught MID-RESTART (`starting`/`unhealthy`), which is how BookStack alarmed on 2026-07-27. The window (`quiesceAlarmGrace` = 180 s, derived from the deploy flow's 120 s health timeout and Mealie's 60 s start_period) **EXPIRES** — permanent suppression turns a loud false alarm into a silent real one. Open-ended while the cycle runs (a first offsite snapshot legitimately takes hours). |
| `agentapi.BackupTiers` / `BackupDueFor` / `StartBackupFor` / `BackupStatusFor` | controller/internal/agentapi/backup_tiers.go | `(ctx[, target]) (…, error)` | The per-tier agent surface (agent >= v0.97.0) | `targetQuery("")` returns an EMPTY suffix so an untargeted call hits the pre-R-82 route byte-for-byte. `BackupTiers` maps a 404 to `ErrTiersUnsupported` — the documented ROUTE-PROBE capability signal, NOT a `featureProbes` row (the loop needs the tier LIST, not a yes/no). |
### Compose ops / stack lifecycle
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `Manager.DeployStack` | controller/internal/stacks/deploy.go | `(req DeployRequest) (string, error)` | Full deploy flow | Sets in-memory `Deployed` BEFORE compose up (slow-pull race), reverts on failure |
| `Manager.RedeployFromEnv` | controller/internal/stacks/deploy.go | `(name, env map[string]string) error` | Re-up with changed env (migration flip, config edits) | `compose up -d`, never `restart` (restart won't pick up images/env) |
| `Manager.PersistUnitRedeployConfig` (R-47, v0.153.0) | controller/internal/stacks/deploy.go | `(name, env map[string]string) error` | the PERSIST half of `RedeployFromEnv` — app.yaml + locked fields + in-memory flags, **starts nothing** | **TRAP: the restore paths must use THIS, never `RedeployFromEnv`.** RedeployFromEnv ends in a full `up -d`, which before the replay IS the H4 race. RedeployFromEnv is now literally this + the unchanged up-and-report tail |
| `Manager.StartStackServices` (R-47, v0.153.0) | controller/internal/stacks/manager.go | `(name string, services []string) error` | scoped `compose up -d <svc>...` — the DB-only window a dump is replayed in | **REFUSES an empty list** (argument-less `up -d` is a FULL start — the one silent fall-through that would reintroduce the race). No `logPostStartStatus`: the app containers are absent on purpose. Never `RestartStack` here — it is a full up in disguise |
| `appbackup.DBServiceNames` / `dbTypeForImage` (R-47, v0.153.0) | controller/internal/appbackup/dbservices.go | `(composePath string) ([]string, error)` | naming the compose SERVICE(s) holding a database, sorted | yaml.v3 `services:` MAP parse — **never a line scan** (immich's top-level `immich_ml_cache:` / `immich_postgres_data:` volume keys look exactly like services). `dbTypeForImage` is shared with `DiscoverDatabases`, which is what makes "a dump exists ⇒ a service can be named" hold. An error means CANNOT-TELL, never "no database" — callers refuse when a dump exists |
| `Manager.StartStack/StopStack/RestartStack/UpdateStack` | controller/internal/stacks/manager.go | `(name string) error` | Lifecycle | Protected stacks refuse stop; all funnel through composeExec. **NOT writers of desired state (R-166)** — 14 call sites, only 2 are the customer; recording intent here would make a nightly backup indistinguishable from the customer pressing Stop. Use `SetDesiredState` at the intent point instead |
| `Manager.SetDesiredState` / `DesiredStateOf` / `BackfillDesiredState` (R-166, v0.189.0) | controller/internal/stacks/desiredstate.go | `(name, desired string) error` / `(Stack) string` / `() int` | THE customer-intent record — `app.yaml``desired_state`, tri-state `""`/`running`/`stopped` | **ONE OWNER: the customer's action.** Writers are the API action switch, `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, and the `.fab` restore adapter — nothing else, ever. **`""` (absent) means UNKNOWN, never "running"**: every pre-v0.189.0 app.yaml reads absent, so treating it as running would start every deliberately-stopped app on upgrade. Write intent BEFORE the act and REFUSE the act if it fails (§8.2). Backfill is **running-only** — never infer `stopped` from zero containers, that inference IS the defect |
| `Manager.DriveLive` (R-171, v0.190.0) | controller/internal/stacks/deploy.go | `(hddPath string) bool` | is an app's data drive a live mountpoint RIGHT NOW | Wraps the **same**`isMountPoint` seam the userdata belt uses (`manager.go`) — never write a second liveness check, the two would drift invisibly. The system/local path is legitimately not a mountpoint and returns true |
| `bootrecon.StartGate` (R-171, v0.190.0) | controller/internal/bootrecon/bootrecon.go | `MayStart(stack) (bool, reason)` | THE one question the boot sweep asks before starting anything | **Fail-safe: cannot determine ⇒ return FALSE.** One seam for all three holders (absent drive · quiesce · an in-flight app-data operation) because they differ only in the reason string. Implemented in `main.go` (`bootDriveGate`) reusing `quiesce.SuppressedStacks()`, `AppStopGuard.HeldStacks()` and `Manager.DriveLive` — never re-derive any of them. Held apps go to `Result.HeldByDrive`, **never**`StillDown` (that is the dead-app alarm's bucket) |
| the boot settle window (R-157 A, v0.190.0) | controller/cmd/controller/main.go | `bootReconcileSample` / `StableFor` / `Budget` | sample the fleet until it stops changing, then sweep ONCE | **settle + budget + one `DefaultRetryDelay` must stay under `deadAppBootGrace`** — pinned by `TestBootWindow_CommonCaseFitsInsideTheDeadAppGrace`, which is why the budget is 50 s and not 60 s. Sampling is READ-ONLY; sweeping per sample would never see a settled fleet (the sweep's own StartStack changes it). A late recovery is REPORTED (`recordLateRecovery`), never hidden by widening the grace |
| `backup.AppStopGuard` (`Begin`/`End`/`Recover`) (R-166, v0.189.0) | controller/internal/backup/appstop_marker.go | `(opID, reason, stacks) error` / `()` / `() *AppStopRecovery` | THE crash marker for stop→work→start windows (volume dump, offbox reconstitute, `.fab` export) | Its **own** file (`appstop-state.json`), never quiesce's — one file, one writer. **A `defer` is NOT the mechanism** (Campaign 8 fault 10: SIGKILL runs no defer); the marker is. Written BEFORE the stop, cleared ONLY after a restart that succeeded; a FAILED restart deliberately KEEPS it. `Recover` RETURNS its outcome rather than notifying, because it must complete before the boot reconciler while the notifier does not exist yet |
| `backup.ErrStartRefused` + `AppStopRecovery.Refused`/`Alarming()` (R-174, v0.191.0) | controller/internal/backup/appstop_marker.go | `errors.Is(err, ErrStartRefused)` / `() bool` | THE refusal-vs-failure split in the app-stop crash recovery | **A gated starter's refusal is NOT a restart failure.**`Recover`'s starter MUST be the gated `gatedAppStopStarter` (cmd/controller/main.go), never the raw `stacks.Manager` — that was the v0.189.0 defect, which started apps onto ABSENT drives at boot (R-171 one path over). A refusal goes to `Refused` (marker KEPT, silent), a real error to `Failed` (marker kept, ALARMS). Collapsing them routes a deliberate hold into `NotifyBackupFailed`, a customer-enabled type — the R-171 false alarm again. `main.go` must guard the notify with `Alarming()`, not `!= nil` |
| `Manager.DeleteStack` / `RemoveStack` | controller/internal/stacks/delete.go | `(name, removeHDDData[, backupPaths])` | THE guarded removal paths | Orphan/protected/deploying/running checks + ProtectedHDDPaths filter before any RemoveAll |
| `resolveContainerState` / `aggregateState` | controller/internal/stacks/manager.go | `(dockerState, dockerStatus)` / `([]ContainerInfo)` | State classification | `.State` says "running" even when unhealthy — `.Status` parse is the fix |
| `Manager.logPostStartStatus` | controller/internal/stacks/manager.go | `(name, stackDir, env)` | Async post-start verification | compose up exits 0 on crash-loops; this is the detection. Goroutine + 3s, never blocks |
| `Manager.EnsureBaseStack` | controller/internal/stacks/infra.go | `() error` | Traefik/cloudflared/FileBrowser infra convergence | Renders from `internal/infra` templates |
| `appbackup.ClassifyBinds` / `ValidateBackupSpec` | controller/internal/appbackup/classify.go | `(spec, binds) ([]ClassifiedBind, bool)` / `(spec, binds) error` | Backup-classification (Task 2, referential coupling) — pure | Two-level default: explicit wins over `:ro`; unlisted writable→mandatory, unlisted `:ro`→excluded; nil spec→legacy/false. Validate REJECTS the WHOLE block on any defect (whole-block semantics). INERT — no tier consumes it yet |
| `ParseComposeClassifiableBinds` | controller/internal/stacks/classify_binds.go | `(composePath) []appbackup.ComposeBind` | `${VAR}`-relative binds + `:ro` for classification | Do NOT use `ParseComposeHDDMounts`/`ExportDataMounts` as classifier input (§traps) — they resolve absolutes, drop `:ro`, or union the userdata ROOT. Short-syntax only |
| `Metadata.EffectiveLifecycle` / `CanInstall` / `IsAbandoned` + `web.lifecycleBadge` / `web.visibleCatalogStacks` | controller/internal/stacks/metadata.go, controller/internal/web/metabadge.go, controller/internal/web/handlers.go | `meta.CanInstall() bool` | app lifecycle: `available` / `hidden` / `abandoned` (v0.158.0) | THE single interpretation of `.felhom.yml``lifecycle:` — every surface must go through these, never compare the raw string. Listing drops `!Deployed && !Protected && !CanInstall()`; `api.deployStack` refuses server-side BEFORE any mutation (hiding a button is not a gate), `stacks.DeployStack` repeats it for non-API callers. **Unknown value fails OPEN** (→ available + one WARN) — opposite to the gate on purpose: a typo must never pull a working app out of every catalog. **NEVER let lifecycle reach orphan detection** (`getCatalogTemplateSlugs`) — a withdrawn template stays in the tree, or every deployed instance reads as `Elavult` and gets a Törlés button. Badges: `MetaBadge` + `meta_badge` partial, built generic for R-56 difficulty labels |
| `Manager.ClassifiedBinds` + `StackDataProvider.GetStackClassifiedBinds` | controller/internal/stacks/metadata.go, appbackup/appdata.go | `(name) ([]appbackup.ClassifiedBind, bool)` | Per-stack classification through the REAL LoadMetadata validate path | The wired seam Task 3 consumes; LoadMetadata is the SINGLE validation choke point (bad block → nil + one ERROR → legacy) |
| `backup.Manager.DumpAppVolumesSafe` | controller/internal/backup/backup.go | `(stackName) error` | Volume tar of a live app | Stops → dumps → restarts; surfaces BOTH errors (app may be left stopped). Check `GetDockerVolumes()!=0` + `IsProtectedStack` BEFORE calling — it stops the stack before its own volume check (see `runVolumeDumps`) |
| `backup.Manager.ListRestorePoints` | controller/internal/backup/restore_points.go | `(stackName) ([]RestorePoint, bool)` | Restorable keep-side backups (the /api/backup/snapshots payload) | ONE point per app (the current unit); tier always 1 — never list Tier-2 (not restorable via /backup/restore) |
| `backup.Manager.RestoreTier2Files` | controller/internal/backup/tier2_restore.go | `(stackName) (filesRestored int, err error)` | In-place ADDITIVE-ONLY class-C file restore from the recorded Tier-2 copy (`POST /backup/tier2/restore`) | Never overwrites/deletes live files; refusals (Hungarian) before any stop; source = recorded `DestinationPath`, never re-selected. **C9-F1 (v0.183.0): reads `hdd/` + `userdata/` ONLY — never `recovery-unit/`.** For 43 of 53 catalog apps that is a guaranteed no-op, so it now refuses with `ErrTier2NoRestorableData` BEFORE stopping the app. Ask `Tier2RestoreCoverage` first |
| `backup.Manager.Tier2RestoreCoverage` | controller/internal/backup/tier2_restore.go | `(stackName) (Tier2Coverage{Legs, HasUnit}, error)` | Answers what a Tier-2 restore CAN and CANNOT return for an app, from the RECORDED copy on disk | **C9-F1.**`Legs` = subtrees the restore reads; `HasUnit` = the copy also holds DB dumps + volume tarballs it will NEVER read. Use it to refuse up front and to decide whether the success message must disclose uncovered data. Judged from the copy, not the catalog, so a retemplated app is judged by what it actually has |
| `Manager.acquireRunning`/`releaseRunning`, `acquireMigrating` | controller/internal/backup/backup.go, controller/internal/stacks/migrate.go | `() error` | Single-flight for long ops | Copy this mutex-flag pattern for any new long-running manager op |
### Secrets hygiene
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `crypto.Encrypt/Decrypt/IsEncrypted/DecryptMap` | controller/internal/crypto/crypto.go | AES-256-GCM, `ENC:` prefix | app.yaml sensitive values | `Decrypt` errors on non-ENC input — use `DecryptMap` for whole env maps (passes through + warns) |
| `Manager.GenerateSecretForField` | controller/internal/stacks/deploy.go | `(stackName, envVar) (string, bool)` | Replacement value for a RESETTABLE secret from its catalog `generate` spec (O4 restore path via `backup.SetSecretGenerator`) | REFUSES `data_key` fields, spec-less and non-secret fields; never log the value |
| `reconcileRestoreSecrets` | controller/internal/backup/restore_unit.go | `(nonSecretEnv, unitSecrets, guestSecrets, secretNames, dataKeyNames)` | Recovery-unit restore env merge | **Precedence: UNIT WINS over guest** (the unit's secrets match the data being restored; the guest's are merely newest). Pure — new sources arrive as ARGUMENTS. Fail-closed data-key gate lives here |
| `stacks.PortableSecretEnvVars` | controller/internal/stacks/deploy.go | `(meta) []string` | **THE D5 secret boundary**: which secrets may travel on a customer drive | `type: secret` travels, `type: password` NEVER, minus the `nonPortableSecrets` code register. Withholding the password class is what licenses plaintext — do not relax one without the other |
| `buildUnitAppYaml` / `readUnitEnv` | controller/internal/backup/{recovery_unit,restore_unit}.go | `(info) []byte` / `(path, portableNames)` | The ONE place the unit's app.yaml is written / split back | Split is driven by the MANIFEST's portable names, never guessed from key names; write 0600; empty `portableNames` = schema-1 unit ⇒ everything is plain config |
| `metrics.RedactLine` | controller/internal/metrics/redact.go | `(s string) string` | ANY log line shipped off-box (issue context, log tails) | Masks password/passwd/secret/token/api-key/authorization/bearer values + 64-hex; apply BEFORE the line leaves the box — controller-side redaction is authoritative |
| `settingsRetrievalPasswordRevealHandler` | controller/internal/web/handlers.go | `POST /settings/retrieval-password/reveal` | **THE PATTERN for showing a secret in the UI** — an XHR that returns only the value | **Never template a secret into a page and hide it with CSS.**`display:none` / `hidden` / `type="password"` stop a browser DRAWING the value; the plaintext is still in the response body, so a `curl` of the page returns it, and it reaches caches, history and any screen-share of the source. R-249 shipped exactly that for two months and was found by it landing in a transcript. The page carries a **boolean** (`HasRetrievalPassword`); the value comes from a POST (CSRF-covered, uncacheable) and the reveal is **logged as an act**. `escrow_handlers.go` states the same rule for R. **Test on the RESPONSE BODY** — a test asserting what the customer *sees* cannot see this class at all. **Both R-254 sites are now FIXED the same way** — `POST /apps/<slug>/initial-credentials/reveal` (re-reads the container, never a cached copy) and `POST /stacks/<name>/auto-field/reveal` (authorised on the field being a `type: secret` auto-field of that stack). **Per-secret, never one generic reveal-any-named-secret endpoint.** The PRE-DEPLOY hidden input is deliberate and untouched — a form must carry what it submits (README §318). Enforced by `scripts/secret_in_markup_gate.py`, whose measured blind spot (a secret under a neutral page-data key) is in its docstring; runtime body-assertion covers 4 of 27 pages — R-255. |
### Storage registry + mount detection
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `Settings.AddStoragePath/RemoveStoragePath/RepointStoragePath` | controller/internal/settings/settings.go | registry CRUD | ALL drive registration | `AddStoragePath` dedupes (double-register is clean no-op); `AutoDiscoverStoragePaths` never re-adds a known-in-any-state path |
| `Settings.SetDisconnected/ClearDisconnected/SetDecommissioned` | controller/internal/settings/settings.go | state flags + stopped-stacks memo | Drive lifecycle state | Records `stoppedStacks` so reconnect restarts exactly those |
| `registerStoragePath` | controller/internal/web/storage_handlers.go | `(where, label, setDefault) error` | Post-enroll registration | The single funnel used by init/attach/manual-add |
| `system.IsMountPoint` / `IsWritable` / `PathsOverlap` | controller/internal/system/mounts_linux.go | `(path) bool` | Mount checks | `_other.go` stubs return permissive values — Linux behavior is the real one |
| `appexport.DiskFree` | controller/internal/appexport/estimate.go | `(path) int64` | Free bytes for space gates (df-based, 0 on any error) | Exported v0.128.0 for the browser-upload gate; test seam = `web.uploadDiskFree` package var |
| `stacks.ExportDataMounts` | controller/internal/stacks/delete.go | `(composePath, hddPath) []string` | THE .fab-export mount discovery (v0.130.0 C6B-F1) | Unions `${HDD_PATH}` binds + the `${USERDATA_PATH}` ROOT (single `userdata` entry — basename must round-trip the import's `<HDD_PATH>/<subdir>` mapping; NEVER return per-bind userdata subpaths). Containment-deduped. Backup-side `stackAdapter` deliberately does NOT use it |
| `Server.deployedAppsOnPath` | controller/internal/web/netstorage_handlers.go | `(base) []string` | Deployed stacks whose HDD_PATH is base or a subpath | The C6B-F2 share-removal guard; nil-safe on stackMgr |
| `planDriveGates` / `Server.ReconcileDriveGates` | controller/internal/web/intermediary.go | pure plan + executor | Drive appear/disappear reactions | `planDriveGates` is PURE (unit-testable); loop at `driveGateLoop` |
| `Server.runStorageInit` / `runStorageAttach` | controller/internal/web/storage_handlers.go | wizard pipelines | New-drive enroll / re-attach | Format goes through the agent's two-step confirm (below) |
| `Server.sharingResolvePath` / `sharingResolveStorageRoot` | controller/internal/web/sharing_handlers.go | `(raw) (string, error)` | THE guard for every customer-supplied SMB share path | resolvePath validates a share TARGET (refuses the drive root); resolveStorageRoot validates the new-folder PARENT (accepts exactly a registered live root). Refusals are UNIFORM (no filesystem oracle). Never add a second deny-list — `stacks.SharingDeniedRoots` derives from `ProtectedHDDPaths` |
### Agent local-API client (cross-repo edge)
| Symbol | File | Short signature | Use for | Gotchas |
| `Server.agentClient` | controller/internal/web/agent_disk_handlers.go | `() (*agentapi.Client, error)` | THE memoized client accessor | Always use this, never a fresh `agentapi.New` per request (§3) |
| `Server.ProbeAgentChannel` | controller/internal/web/agent_disk_handlers.go | `(ctx) (constructionErr bool, err error)` | Channel health probe | Probes via the PRODUCTION client on purpose (self-heals, mirrors UI) |
| `Client.EjectDisk` / `Decommission` / `AssignDisk` / `GuestAttach` / `ListCandidates` | controller/internal/agentapi/client.go | disk lifecycle | Delegate ALL disk ops to agent | Controller holds no Proxmox creds — never shell out to disk tools in-guest |
| `Client.AddNetStorage/ListNetStorage/RemoveNetStorage` | controller/internal/agentapi/client.go | NAS mounts (A1) | Network storage | Password passes through to agent's 0600 cred file; controller NEVER persists it |
| `agentapi.StatusError` | controller/internal/agentapi/client.go | `{Path, Code}` typed non-2xx GET error | Distinguishing HTTP statuses from transport errors (`errors.As`) | NEVER string-match agent error text — the capability probe keys on `Code==404` |
| `SupportCache.Supports` / `Client.Supports` | controller/internal/agentapi/features.go | `(ctx, prober, Feature) SupportState` | Agent-capability gate for COUPLED features (route probe, TTL 5m) | 404 ⇒ No; transport/5xx ⇒ Unknown (NEVER refuse on Unknown). New coupled feature = new `featureProbes` row + gate call at the entry point + `MinAgent:` in the CHANGELOG header (publish-train-rules.md). Web layer: `Server.netFeatures` through the `netAgent` seam |
| `agentapi.DiskVerdictFor` / `DiskVerdict.Label` / `DegradedAttributes` / `UncorrectableSectors` / `DiskPrior` / `TemperatureFailC` | controller/internal/agentapi/diskverdict.go | `(*SmartSummary, DiskPrior) DiskVerdict` | THE shared disk-health verdict (card chip + hourly check) — v0.169.0, 14-row ladder v0.215.0 | Pure — no clock, no I/O; history arrives as `DiskPrior`. nil/UNKNOWN → `DiskVerdictUnknown` (Nincs adat, NEVER alarms, row 1 is first for that reason). **Never trust `smart_status.passed`**: attrs 187/197/198 carry `thresh: 0`, so it cannot fail on unreadable sectors. A zero `DiskPrior` is the fail-safe (first sighting can only reach Figyelmeztetés). **Four labels, no fifth** — predicted failure is „Hiba". Do NOT recompute the verdict inline anywhere, and do NOT re-literal 60 °C — use `TemperatureFailC` |
| `Server.resolveBackupTargetState` / `backupTargetView` | controller/internal/web/backup_target_offer.go | `(ctx)` → state / `*BackupTargetView` (nil = render nothing) | The whole-system backup-target answer: healthy · degraded-never-configured · **TargetAbsent** (configured, drive gone) · unknown | Test seams `Server.tiersFn` + `Server.disksFn` (nil → the real client). **`degradedMessageFor` is the ONE place that decides customer copy** — add a state there, never in a template. `backupTargetView` returns **nil** for healthy AND unknown so a template typo cannot decorate a working box. R-112: this state had NO consumer for two releases; the render is server-side on `backups.html`, and the seam test drives `backupsHandler` and asserts rendered HTML |
| `Server.cachedDisks` / `RunDiskHealthCheck` | controller/internal/web/disk_health.go | `(ctx)` | Card fetch (60s TTL) / the hourly degradation check | Card uses the 60s TTL cache (anti-smartctl-storm); the CHECK fetches FRESH (`fetchDisks`). Test seams: `Server.disksFn` (source) + `Server.diskNotifyFn(notify.DiskAlert)` (sink). State is PERSISTED (v0.215.0) — a restart no longer re-baselines |
| `diskAlertDecision` / `diskAlertKindFor` / `Server.priorFor` / `Server.cardPriorFor` | controller/internal/web/disk_health_state.go | pure + `(key) agentapi.DiskPrior` | Whether an observation emits, and which message shape | Compares against the **last ALERTED** verdict, not the last observed — that is what collapses a flap to one alert. Re-alert needs doubling **AND** 24h (an AND). **`priorFor` is for the CHECK, `cardPriorFor` for the CARD** — they differ by one observation and mixing them makes the chip read one level more severe than the email |
| `diskRecord` / `writeDiskState` / `Server.loadDiskStateLocked` | controller/internal/web/disk_health_state.go | `disk-health-state.json` in `cfg.Paths.DataDir` | Persisted per-disk observation + alert history | Atomic tmp+rename (the `selfupdate.SaveState` shape, copied not imported). Missing file = normal; corrupt = LOG and fall back to no-prior, **never fatal**. Written ONCE per check run. Keyed by `diskKey`. **One record per disk, NOT a sample series** — history is Phase 2/3 in `metrics.MetricsStore` |
### Notifications / hub sync
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `Notifier.PushEvent` | controller/internal/notify/notifier.go | `(eventType, severity, message, details)` | Hub events | Async goroutine, 3 attempts/3s backoff. NEW event types MUST be added to hub `allowedEventTypes` or POST /event 400s. **SEVERITY IS AN EXACT WIRE CONTRACT: `{"info","warning","error","critical"}` and nothing else.** The hub silently COERCES any other string to `"info"` (`hub/internal/api/handler.go`, the ingest severity switch) and `severityNotifies` (`hub/internal/notify/dispatcher.go`) emails only warning/error/critical — so a typo'd severity is stored and delivered to NOBODY, with no error anywhere. **`"warn"` is not a severity.** It shipped on `disk_health_degraded` (fixed v0.215.0, R-328) and is STILL live on `app_start_failed` (R-329) |
| `notify.DiskAlert` / `DiskAlertKind` / `DiskAlertKind.Severity()` | controller/internal/notify/notifier.go | `NotifyDiskHealthDegraded(DiskAlert)` | The disk-health alert payload + its five Hungarian message shapes | The notifier owns customer copy — pass a `DiskAlert`, never a pre-formatted string, or Hungarian scatters across packages. `Severity()` is the ONE mapping kind→hub severity and is exported so any package can assert the contract instead of duplicating the literal |
| `Notifier.Notify*` convenience methods | controller/internal/notify/notifier.go | typed wrappers (backup/DB/storage/channel/DR…) | Standard events | Add a typed wrapper rather than raw PushEvent calls |
| `report.Trigger` (`NewTrigger`/`Fire`/`Run`) | controller/internal/report/trigger.go | `Fire()` after a hub-relevant user action | THE out-of-cycle report push (v0.139.0) — fire via `api.Router.reportPushNow` / `web.Server.reportTriggerNow`, both nil-safe | Coalesce-and-eventually-fire (trailing edge; quiet 2s, min spacing 15s). NEVER add retries (Pusher owns them); NEVER reuse the `internal/sync` REFUSE-debounce for hub pushes (a refused fire loses the update until the next cycle). Fire only AFTER a successful local commit |
| `report.SetPendingLogTails` + `buildLogTailsSection` | controller/internal/report/logtail.go | ACK `log_tail_requests` → next report `log_tails` | THE pull-based ACK-flag pattern (hub asks, controller pushes next cycle) — copy for any new hub→box request | Consume-once drain at BuildReport; failed push re-arms from the hub's still-pending request; NEVER add a hub→controller push channel |
| `offsiteapply.SettleProvider` / `SettleFunc` / `Bridge.AwaitSettle` / `ReconcileWhenSettled` (R-71a, v0.162.0) | controller/internal/offsiteapply/offsiteapply.go + seams.go | `SettleState() (version, floor string, updateRunning, floorKnown bool)` | THE settle-gate: defers the offsite one-time-password consume past a managed day-0 floor-update (the F10 race). Wire the `SettleFunc` adapter over `updater.GetFloor()`/`IsUpdateRunning()` — **the updater's knowledge is the ONE floor source; never fetch the floor a second way**. Gate ONLY the bridge goroutine, and only when an updater exists (nil `Settle` = reconcile immediately). Bounds `settlePoll`/`settleFloorSubBound`/`settleOverallBound`; the floor is in-memory (report-ACK-derived, ~5–10 s), NOT persisted → unknown until the first ACK on any restart. Inject `Now`/`Sleep` in tests (no real sleeps). B′: at/above-floor GOes on the first poll, zero wait. Do NOT touch the consume/persist order or the 404 contract — ordering only |
| `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it |
| `Manager.beginRunSummary` / `noteFailure` / `noteAttempted` / `emitRunSummary` / `SetRunSummaryNotify` (R-182, v0.194.0) | controller/internal/backup/runsummary.go | `(kind, runID) func()` / `(app, leg, reason)` / `(RunSummary)` | **THE per-run operator digest.** One `backup_run_failures` event at the end of a run listing every failed app, its leg and its reason — emitted ONLY when something failed | **The RECORD and the NOTIFICATION are different things and must stay so.** The per-app `recovery_unit_capture_failed` event is the record (hub routes it *record-only*, stored + logged every time); this digest is the notification. Before R-182 one event was both, and did neither: nine arrived, two were mailed, seven vanished before `LogNotification`. **Lifetime is `admissionSet`'s exactly** — absent collector means "no run in flight", never a stale answer. **A refusal is noted ONCE, inside `admitApp` where the verdict is taken**, not at the three legs that consult it: R-181's one-verdict-covers-all-three contract makes per-leg noting produce "2 of 1 apps failed". **Deliberate skips (disconnected / decommissioned) must NEVER be noted** — they have their own alert and a nightly digest about an unplugged drive is an ignored digest. **A clean run emits NOTHING**; silence is safe only because the hub's deadline check (`monitor/deadline.go:396,417`) raises a missed backup from report freshness independently — if that is ever weakened this design loses its footing. **`run_id` is unique per real run** (so the hub's 1-h cooldown cannot collapse a manual run into the nightly one) and **deliberately EMPTY on the periodic refresh sweep**, which must stay under that cooldown or a polled status page becomes a mail flood |
| `Manager.admitApp` / `beginAdmissionRun` / `decideAdmission` / `estimatedWriteBytes` (R-181, v0.193.0) | controller/internal/backup/admission.go | `(stackName) bool` / `() func()` | **THE reserve gate. Call it before ANY per-app backup write** — one verdict per app per run, covering the DB dump, the volume dump and the unit capture (all three write under one per-app root) | **Decided LAZILY at the app's first write, never once at run start** — app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. **Never re-decided between an app's own legs**: that is exactly the split R-181 closed (bulk written, capture refused). **Reset per run** via the closer `beginAdmissionRun` returns. **Must sit ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it has already bounced the app. Fires **exactly one**`unitNotify` per refused app per run. Nil admission set (periodic status refresh) → decides fresh, which is still once per app per sweep. Wiring pinned by an **AST walk** in `TestAdmission_IsWiredIntoEveryProductionWriteLeg`, not `strings.Contains` |
| `Manager.floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` / `floorReason` (R-165 B2 v0.192.0, size term R-181 v0.193.0) | controller/internal/backup/recovery_unit.go | `(*UnitSpace, estGiB float64) (*UnitSpace, floorReason)` | The pure two-question predicate behind `admitApp`: is the filesystem already below the reserve (`floorHeadroom`), and would THIS app's write take it below (`floorSize`)? | **Headroom is about the FILESYSTEM, never a per-unit cap** — a size cap is R-163 rebuilt inside one volume; the size term bounds the *delta*, not the unit. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. **`estGiB == 0` degrades to headroom-only on purpose** — refusing an app with no history makes the FIRST backup the one that can never happen. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk |
| `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)` → `Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing**`disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space |
### Scheduler / time / UI
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `Scheduler.Every` / `Daily` | controller/internal/scheduler/scheduler.go | `(name, interval/"HH:MM", fn)` | ALL background jobs | Daily is Europe/Budapest, DST-safe (`nextDailyRun` avoids Add(24h)); register in main.go block (§5) |
| `Scheduler.UpdateDaily` | controller/internal/scheduler/scheduler.go | `(name, "HH:MM") bool` | Retime a daily job at runtime (no restart) | Per-job buffered `resched` chan + select case in `runDailyJob`; false (WARN) on invalid time / unknown-or-non-daily name; read `Schedule` under the mutex in the loop |
| `getBudapestLocation` | controller/internal/scheduler/scheduler.go | `() *time.Location` | Local-time math | web has its own `getTimezone` (§6); quiesce has its own `budapestLocation` (window gate) — 3rd copy, see §6 |
| `Server.templateFuncMap` | controller/internal/web/funcmap.go | template.FuncMap | ALL template functions | `stateColor` outputs v2 suffixes `run/progress/warn/neutral/off`; stopped = NEUTRAL not red (operator-approved); `stateLabel` copy is frozen byte-identical (unit-tested) |
| `timeAgoStr` | controller/internal/web/funcmap.go | `(s RFC3339 string) string` | Ago-format for STRING timestamps | Exists because `timeAgo(time.Time)` 500'd on strings (v0.93 bug) |
| `Server.RequireAuth` / `CsrfProtect` / `csrfField` | controller/internal/web/auth.go + csrf.go | middleware | Any new authed route/form | csrfField emits the hidden input; setup wizard has its OWN csrf (§6) |
| `LogBuffer` + `Lines(maxBytes)` | controller/internal/web/logbuffer.go | ring buffer io.Writer | In-memory log capture for the debug UI + the report `controller_log_tail` source | v0.116.0: ALWAYS constructed (any logging.level) — the logger is `MultiWriter(LevelFilterWriter(stdout, level), ring)`; `Lines` drops OLDEST to honor the byte budget |
| `logx.Debugf/Infof/Warnf/Errorf` | controller/internal/logx/logx.go | `(l *log.Logger, format, args…)` | ALL NEW leveled log lines (the v0.116.0 sweep standard) | routing is the WRITER's job — Debugf always reaches the ring, stdout filters; nil logger = no-op; caller-attributed (Output calldepth 3) |
| `web.LevelFilterWriter` | controller/internal/web/levelfilter.go | `NewLevelFilterWriter(w, minLevel)` | stdout leveling under the always-on ring | untagged lines parse INFO; always reports full length written |
| `monitor.RunHealthCheck` / `EffectiveProtected` | controller/internal/monitor/healthcheck.go | system health report | Health + protected-container list | — |
| `util.TruncateStr` | controller/internal/util/strings.go | `(s, maxLen) string` | Rune-safe truncation | The intended shared helper; stacks still uses its byte-based twin (§6) |
## 2. Canonical patterns (copy structure from THE named file)
| Two-step confirmed destructive op | controller/internal/web/storage_handlers.go `handleStorageWipe` | server-side type-to-confirm + probe(unconfirmed) → sentinel error → resubmit bound to agent durable-id; agent re-checks role regardless |
| Crash-safe long job (journal + recover) | controller/internal/stacks/migrate.go | state machine + `writeJournal` per transition + `RecoverMigration` at startup + single-flight acquire/release + done-hook |
| Quiesce/marker loop | controller/internal/quiesce/quiesce.go | marker BEFORE side effects, guaranteed undo (defer + max bound), `Recover()` once at startup, `TriggerNow` 409 single-flight |
| Platform split | controller/internal/system/mounts_linux.go + mounts_other.go | `_linux.go`/`_other.go` twins; other = permissive no-op stubs for dev on Windows |
| Debounced trigger + status (REFUSE-style — a too-soon fire is refused/lost) | controller/internal/sync/sync.go | `TriggerSync` 30s debounce, `Status()` snapshot struct, post-sync hook fan-out |
| Coalescing trigger (trailing edge — a burst collapses but the LAST state always fires) | controller/internal/report/trigger.go | buffered-1 chan + non-blocking `Fire()` + single worker (quiet window → drain → min-interval → fire once); shape from hub `wgsync/reconciler.go` |
| Detached job + status poll (single-flight, phase strings) | controller/internal/web/storage_init_job.go | acquire/release/set/**deep-copied** snapshot; phases mapped to Hungarian in the template; 1–3 s poll; terminal state **PROBED, not inferred**. Clones: `netstorage_job.go`, `samba_ensure_job.go` (v0.147.0). **Five of these now exist and agree on nothing — R-45 will unify them; prefer extending an existing one over a sixth** |
| Streaming subprocess progress | controller/internal/backup/offbox_progress.go | `offboxStreamRunner` seam (stdout scanned line-by-line, stderr buffered, output tail-bounded) + a PURE line parser + a mutex-guarded published snapshot. Traps it encodes: a source reporting nothing is **normal** (restic sends 0 bytes for a whole incremental run) and the progress source may only update on unit completion — degrade bytes → files → current item + elapsed, never fake a percentage |
| Post-start async verification | controller/internal/stacks/manager.go `logPostStartStatus` | goroutine + sleep, INFO log, never blocks/fails the operation |
| Startup wiring order | controller/cmd/controller/main.go | init-only setters (`SetStackProvider` M2 contract: exactly once, before scheduler/HTTP), scheduler registration block |
## 3. Dangerous lookalikes — do NOT reuse
| Trap | Why it bites | Use instead |
|---|---|---|
| `rsyncMirror` (controller/internal/backup/tier2.go) | `rsync -a --delete` — DESTROYS anything extra at dst; correct only for tier-2 mirror dirs (backup DIRECTION). In the tier2→live restore direction it would erase every live file created since the last copy | `rsyncCopy` + `rsyncVerify` (controller/internal/stacks/migrate.go) for any move/copy; `rsyncRestoreMissing` (controller/internal/backup/tier2_restore.go, `-a --ignore-existing`) for the additive-only restore direction |
| raw `os.RemoveAll` on drive/HDD paths | Bypasses the protected-set; wipes appdata/backups/media | `Manager.DeleteStack`/`RemoveStack` (controller/internal/stacks/delete.go) — gated by `ProtectedHDDPaths` + orphan/protected/running checks |
| `timeAgo` on an RFC3339 string field | Template 500 (OffboxTarget.LastRun bug, fixed v0.96.0) | `timeAgoStr` |
| `backup.Manager.DumpAppVolumes` on a running DB app | Inconsistent tar of live DB volume | `DumpAppVolumesSafe` (stop → dump → restart, both errors surfaced) |
| `stacks.Manager.execCommand` / `composeExecCustomEnv` for NEW long-running calls | No context/timeout — a hung docker CLI blocks forever | `exec.CommandContext` + explicit timeout (copy `rsyncCopy` or appexport `composeExecEnv`) |
| `config.LoadPermissive` | Skips validation — setup-mode only (customer.id/domain may be unset) | `config.Load` everywhere else |
| `ExportDataMounts` / `ParseComposeHDDMounts` as **backup-classification** input | `ExportDataMounts` unions the `${USERDATA_PATH}` ROOT (export-capture logic, not per-bind); `ParseComposeHDDMounts` resolves absolutes AND drops the `:ro` flag — classification needs `${VAR}`-relative paths + read-only awareness | `ParseComposeClassifiableBinds` (controller/internal/stacks/classify_binds.go) |
| `docker compose restart` (any wrapper) | Does not pick up new images or env | `RedeployFromEnv` / composeExec `up -d` |
## 4. Seams & interfaces (testing + cross-repo)
| Interface | Defined in | Implemented by | Fakes/tests at |
| `netAgent` + `Server.netAgentFn/netProbeFn/netListFn` | controller/internal/web/netstorage_job.go (+ server.go fields) | `*agentapi.Client` / `runNetProbe` (linux re-exec) / `agent.ListNetStorage` | `fakeNetAgent` + fn injections in controller/internal/web/netstorage_job_test.go — the NAS add orchestration never shells/TLS-dials in tests |
| `Server.agentLogsFn` (func seam) | controller/internal/web/server.go | nil → `agentClient().DebugLogs` (agent GET /debug/logs) | injected in controller/internal/web/observability_test.go (incl. the pre-0.83 typed-404 notice path) |
| `escrowAgent` + `Server.escrowAgentFn/escrowStageFn/escrowStaleFn` | controller/internal/web/escrow_handlers.go (+ server.go fields) | `*agentapi.Client` / `PushOffboxPasswordForEscrow` / `report.EscrowAutoConfirmer.StaleBlob` (SetEscrowStale) | `fakeEscrowAgent` + fn injections in escrow_wizard_test.go — call-ORDER assertions (stage BEFORE trigger) + agent-never-called gates. The claim leg is the ONLY surface R crosses: no-store, never logged, never templated |
| `offboxCeremonyWaitState` + `escrowCeremonyGraceWindow` | controller/internal/web/handlers.go | pure pick: (awaiting, timedOut) from `OffboxTarget.{EscrowState,CeremonyCompletedAt}` — the v0.138.0 "megerősítésre vár" card. Stamp SET on claim (escrow_handlers.go), CLEARED on the flip (main.go Flip + offbox_handlers.go manual confirm) | escrow_wait_state_test.go truth table (escrowed/unstamped/unparseable → plain CTA; boundary via `>=`) |
| `Manager.sambaUpFn` / `sambaPasswdFn` / `sambaRunFn` / `sambaAddrFn` (func seams) | controller/internal/stacks/manager.go (fields) + samba.go | nil → `composeUp` / `docker exec smbpasswd` (STDIN) / `containerRunning("felhom-samba")` / `docker exec felhom-samba ip -4 -o addr show eth0` | injected in controller/internal/stacks/samba_test.go — the idempotency test asserts the up-seam is called **zero** times when config is unchanged; the passwd seam means no unit test ever handles a real secret or touches docker. **`sambaRunFn` has an EXPORTED setter (`SetSambaRunProbe`)** — internal/web's status-contract tests need a live-container world from another package. `sambaAddrFn` backs `SambaLANAddress()` (v0.151.0); its parse is separately pinned in samba_lanaddr_test.go and it returns "" on any failure — the page omits a line rather than printing a wrong address |
| `Manager.SambaLANAddress()` | controller/internal/stacks/samba.go | `() string` — the guest's LAN IPv4 for the Megosztás connect card (v0.151.0, S-2) | Read from the SAMBA container's netns (`network_mode: host`), never `net.InterfaceAddrs()` — the controller is on a docker BRIDGE and would answer 172.x (the same trap `setup.DetectLocalIPs` needs `HOST_IP` for). **NEVER cache/persist it** — the guest holds it by DHCP (S-5); callers re-derive per render. `""` = omit the line |
| `Server.sambaAddrFn` (func seam) | controller/internal/web/server.go (field) + sharing_handlers.go `sambaLANAddress()` | nil → `stackMgr.SambaLANAddress()` | The web-side half of the connect card. Tests inject a COUNTED fn — the fresh-per-render assertion is what stops anyone memoizing a DHCP lease |
| `Manager.guestNetExecFn` (func seam) + `GuestGateway()` / `GuestNetSnapshot()` | controller/internal/stacks/manager.go (field) + guestnet.go | nil → `docker exec felhom-samba <args>` — ONE seam for all R-66 guest-netns reads (route/link/addr/resolv.conf); tests script canned outputs per argv | guestnet_test.go. **The netns door rule:** the controller's OWN netns is the docker bridge, so any in-process read (`net.Interfaces`, `/proc/net/route`, its own `/etc/resolv.conf` = 127.0.0.11) is the S-2 wrong answer — guest-net reads MUST go through the samba (`network_mode: host`) exec door. Megosztás off ⇒ door closed ⇒ "" / per-item error strings; NEVER substitute an in-process value. Same S-5 law as SambaLANAddress: live per render, never cached/persisted. Parsers (`parseDefaultRoute`, `parseGuestInterfaces`, `parseResolvConf`) are pure + separately pinned |
| `buildFileBrowserPaths` + `fbPathDeps` (R-67, v0.160.0) | controller/internal/web/handlers.go | pure assembly of one FileBrowser sync pass: (mount lines, config source paths) from the registry, with per-kind gates | filebrowser_network_test.go. **Two storage classes, two DIFFERENT gates:** drives keep the drive-absent gate + userdata scoping + skeleton (byte-identical to pre-R-67 — tested); network shares bind the share ROOT `:rslave` with the STUB gate instead (`classifyFSPath`; stub ⇒ excluded from mounts AND sources — an exposed stub swallows uploads the real mount later shadows; idle autofs / unknown ⇒ include, fail open). NEVER call `EnsureUserdataSkeleton` toward a network path (red-proven); never force-wake an idle trigger in the sync (doctrine) |
| `Settings.RefuseAsAppNamespace` (R-108, v0.187.0) | controller/internal/settings/settings.go | `(path) (refuse bool, hungarianReason string)` — may an app's DATA NAMESPACE live here? | **THE single predicate for every placement surface** (deploy POST `api/router.go`, per-app migrate list + `handleStorageMigrateApp`, `handleStorageDecommission` mode=migrate TARGET). **Network storage is refused** because an app's namespace root IS its backup root (`namespaceRoot` returns a non-system drive path as-is → `<HDD_PATH>/backups/primary/<stack>/`), and on a share that lands inside FileBrowser's share-ROOT `download:true` bind — which CANNOT be narrowed (R-67 `:rslave` = automount wake; and apps on a share store at `<share>/<app>`, so there is no `userdata/` to scope to and creating one would write Felhom convention onto a customer's NAS). **DISTINCT from `refuseNetworkLifecycle`** — that asks "may a DRIVE lifecycle op run on this path" and is applied to the op's SUBJECT; this asks "may an app live here" and is applied to a placement TARGET. Migrate needs BOTH. **FAILS CLOSED:**`/mnt/felhom-drives` holds both kinds, so a path prefix cannot classify — `Kind` exists only on a REGISTERED path, therefore an unregistered path under that root is un-classifiable and REFUSES. Empty path = SSD-resident = allowed; nil receiver refuses. network_app_namespace_test.go, 4 red-proofs |
| `Server.guestGatewayFn` / `guestNetFn` (func seams) | controller/internal/web/server.go (fields) + sharing_handlers.go accessors | nil → `stackMgr.GuestGateway` / `stackMgr.GuestNetSnapshot` | network_card_test.go — the counted-fn freshness test (2 renders ⇒ 2 resolves) is what stops anyone memoizing a DHCP lease; the Hálózati név row is gated on `smb.Enabled` (red-proven: gate dropped ⇒ \\FELHOM rendered while samba is down) |
| `sambaEnsureState.consumeIfRunning()` | controller/internal/web/samba_ensure_job.go | serve-once `snapshot()` for terminal `running` only | `/sharing/status` carries a job EDGE (`phase`) and a service LEVEL (`running`) in one envelope — never let a level reach the phase channel, and never re-serve a consumed edge: the client answers `phase=="running"` with `location.reload()`, so both mistakes produce an infinite page reload (S-1/S-4, DIAG-sharing-2026-07-20.md). `failed`/`needs_password`/in-flight are NOT consumed |
| `infra.SambaHostInterface` | controller/internal/infra/samba.go | the guest LAN nic name (`eth0`) | Single source for smb.conf's `interfaces =`, the container's `FELHOM_IFACE`, and the LAN-address read — if they name different nics, the service and the address the page prints drift apart |
| `Manager.sambaImgFn` (func seam) | controller/internal/stacks/manager.go (field) + samba.go | nil → `docker image inspect <infra.SambaImage>` | drives the 4b card's pulling-vs-starting decision, which MUST be taken before `compose up` (afterwards the image is always present) |
| `Manager.offboxStreamRunner` + `SetOffboxStreamRunner` | controller/internal/backup/offbox_progress.go | nil → `defaultOffboxStreamRunner` (real `restic`, stdout scanned live) | streaming sibling of `offboxRunner`; fakes emit canned `--json` status lines in offbox_progress_test.go, so the whole progress path runs with no restic, network or repo |
| `Manager.offsitePreDumpFn` + `SetOffsitePreDumpFn` (R-44, v0.148.0) | controller/internal/backup/offbox_reconstitute.go (seam) + offbox.go (call site) | nil → `runDBDumpsInternal` under the SAME running flag | THE dumps-before-capture ordering seam. Extracted so the order is observable without Docker/restic — an ordering guarantee no test can see is one refactor from silently reverting to the DIAG-immich-restore-2026-07-19 behaviour. Red-proof: moving the capture first yields `[capture dump]` |
| `Manager.offboxFullPlaceCopier` + `SetOffboxFullPlaceCopier` (R-43) | controller/internal/backup/offbox_reconstitute.go | nil → `rsyncRestoreOverwrite` (`-a --itemize-changes`; **no**`--ignore-existing`, **no**`--delete`) | **TRAP: do NOT reuse `offboxPlaceCopier` here.** The two copiers have OPPOSITE semantics for an existing file — `--ignore-existing` is exactly what a full restore must not do, and conflating them is how a missing-only merge came to be labelled a restore. Never `rsyncMirror` (`--delete`) in any restore direction |
| `Manager.safetyDumpFn` + `SetSafetyDumpFn` (R-43) | controller/internal/backup/offbox_reconstitute.go | nil → `DumpOne` | the pre-restore undo. Invariant: the `pre-restore-`-prefixed dump must be verified ON DISK before anything is stopped/overwritten/replayed; failure ⇒ refuse with zero changes. Red-proof requires removing BOTH guards (the `err != nil` return and the `os.Stat`) — removing one leaves the other holding |
| `reimportDBDumpsFrom(ctx, stack, dumpDir)` | controller/internal/backup/restore_db.go | explicit-dir sibling of `reimportDBDumps` (which passes `AppDBDumpPath`) | offsite reconstitution replays from the SCRATCH unit: the live unit is deliberately never overwritten, so replaying from it would replay the current DB over itself and restore nothing |
| The DB-only replay window (R-47, v0.153.0) | controller/internal/backup/{offbox_reconstitute,restore_unit}.go | both restore paths: stop → place/volumes → `StartStackServices(dbServices)` → replay → `StartStack` (full) | **THE ordering invariant.** Replaying while the whole stack is up lets the app's own schema management race the dump — measured at 2 s on 2026-07-19 (H4), replay aborted `already exists`. Fail-closed: a dump with NO identifiable DB service refuses BEFORE the first mutation. Every exit from the window (replay error, DB-only start error) MUST still do a best-effort full start, or a failed restore becomes an outage. `hasReplayableDump` excludes `pre-restore-` safety dumps — counting them would arm the window for an app with nothing to replay |
| `Manager.OffsiteScratchPair` / `OffsitePairInfo` | controller/internal/backup/offbox_reconstitute.go | reads the restored scratch unit's manifest (`offsite_run_id` / `dumps_at`) + the R-44 sniff | the confirm-dialog honesty surface. All warn-level: a pre-v0.148 (unstamped) pair and an empty-looking dump are SURFACED, never blocked — a false positive that refused a legitimate restore would be worse than the skew |
| `appbackup.DumpValidation.LooksEmpty` (R-44 sniff) | controller/internal/appbackup/dbdump.go | computed in ValidateDump's existing single pass; `userTableNames` is EXACT-match | size and table count are both useless as emptiness heuristics (the 2026-07-19 dump: 52MB, 60+ tables, zero users — all geodata). **TRAP: never widen to a substring match on "user"** — it would flag `user_metadata` / `album_user` / `user_audit` on every healthy single-user box. A row wider than the read buffer still counts as a row |
| `Manager.execFn` (func seam) + `restartPolicyLookup` / `inspectRestartPolicyFn` (R-51, v0.156.0) | controller/internal/stacks/manager.go | nil → real `exec.Command` / `docker inspect -f {{.HostConfig.RestartPolicy.Name}}` | `scriptedDocker` in controller/internal/stacks/degraded_test.go drives the WHOLE production path (docker ps → aggregateState → docker inspect) — an aggregateState-only test proves the function, not the caller. Policy answers are cached per container+state and pruned to the live `docker ps` set; a FAILED inspect is deliberately never cached (a hiccup must not pin a container to "unknown") and reads as SUPERVISED, i.e. fail-closed — the opposite of `IsDownState`'s fail-open, because there the state is ambiguous while here a member is known dead |
| `bootrecon.StackProvider` (R-52, v0.156.0) | controller/internal/bootrecon/bootrecon.go | `*stacks.Manager` (GetStacks/StartStack/RefreshStatus) | `fakeStacks` counts StartStack per app; the load-bearing assertion is the NEGATIVE — a zero-container stack (a UI Stop = `compose down` = containers removed) must record **0** starts, while a boot orphan (containers present, Exited) records exactly 1. `Reconciler.sleep` is injected so the 30 s gap costs nothing |
| `bootReconcileFn` + `runBootReconcile` (package-main seam, v0.156.0) | controller/cmd/controller/main.go | `bootrecon.New(mgr, logger).Run` | controller/cmd/controller/bootrecon_wiring_test.go. **The wiring itself is asserted by an AST walk** over `func main()`, not a `strings.Contains` — the substring version passed its own red-proof because a commented-out call still contains the string. Comments are not callers |
| `classifyRunStates` (pure fix-3 derivation, v0.164.0) | controller/cmd/controller/main.go | `([]stacks.Stack, quiesced, failedRestart map[string]bool, now time.Time)` → `(dead []web.DeadApp, states []notify.AppRunState)` | classify_runstates_test.go. **THE single fix-3 rule: down = `(IsDownState(st.State) || st.CrashLooping(now)) && !userStopped && !quiesced`.** C9-F2 (v0.183.0) added the crash-loop term: `restarting` is NOT in `IsDownState` and must not be — adding it alarms on every deploy and update fleet-wide — so a SUSTAINED restarting run (`stacks.crashLoopAfter` = 5 m, above the 120 s deploy timeout, Mealie's 60 s start_period AND R-97b's 180 s grace) becomes down instead. `now` is injected so the threshold is a testable contract. A deliberate UI stop (`compose down` → zero containers → StateStopped, I1) must not alarm — banner OR email — while faults (Exited/Degraded) alarm byte-identically; I2 (P2 census: all catalog services `unless-stopped`) is why a crash never rests at stopped. **Do NOT touch `IsDownState`** (other callers rely on stopped=down) and do NOT filter in `buildDeadAppAlerts`/`NotifyAppStartFailures` — one derivation point. If I1 or I2 changes, revisit the suppression |
| `report.SetPendingControllerLog` / `SetControllerLogSource` | controller/internal/report/selftail.go | ACK-armed consume-once self-log pull (the logtail.go shape) | selftail_test.go; source = `logBuffer.Lines`, wired once in main.go |
| `util.ParseVersion` / `util.Version.Compare` | controller/internal/util/version.go | THE one semver comparator (house rule: never a second) — selfupdate aliases it; agentapi's MinAgent comparison uses it | rejects pre-release/dev/latest (callers fall back, never trust); numeric compare (0.100 > 0.81) |
| `agentapi.AgentVersionReporter` + `featureMinAgent` | controller/internal/agentapi/features.go | version-first Supports (v0.82.0 header channel); probe = fallback for header-less agents | a coupled feature adds BOTH a featureProbes row AND a featureMinAgent row; v0.116.0: `SupportsWithSource` also reports HOW the verdict was reached (version/probe-cache/probe) for the gate log line |
| `netProbeReadBack` (package var) | controller/internal/web/netprobe.go | `os.ReadFile` | overridden in TestNetProbeChild (nonce-tamper + cleanup-fail rows); package var because the child is a RE-EXEC'd process in production |
-`controller/internal/agentapi/client.go` ↔ **felhom-agent** local API (`/storage`, `/disks*`, `/backup*`, `/netstorage*`, `/guest/*`): pinned leaf SHA-256 + per-guest bearer token from bootstrap.json.
-`controller/internal/report/pusher.go` → **hub**`/api/v1/report` ingest; ACK `config_version` drives config_refresh.go; `controller/internal/notify/notifier.go` → hub `/api/v1/event` (hub-side `allowedEventTypes` allowlist must include new types).
-`controller/internal/sync/sync.go` ← **app-catalog-felhom.eu**: copies ONLY `docker-compose.yml` + `.felhom.yml` per app (SHA-256 change detection); NEVER overwrites `app.yaml` (deployed secrets).
## 5. Extension points (where new features plug in)
- **New storage web endpoint**: switch in `ServeStorageAPI` (controller/internal/web/storage_handlers.go); disk ops in `ServeDiskAPI` (controller/internal/web/agent_disk_handlers.go); backup in `ServeBackupAPI`; export in `ServeExportAPI`; debug in `handleDebugAPI` (debug-mode gated).
- **New REST endpoint**: path dispatch in `Router.ServeHTTP` (controller/internal/api/router.go); use `writeJSON` + `limitBody`.
- **New background job**: `sched.Every`/`sched.Daily` registration block in controller/cmd/controller/main.go.
- **Docker volume tar streaming (v0.125.0)**: `appexport.dockerExec` (seam, package var) + `withVolumeHelper`/`exportVolumeTar`/`importVolumeTar` — stream volume content via `docker cp` through a stopped helper container. NEVER `docker run -v <controller-local path>` — the daemon resolves `-v` host-side and strands the data when the controller is containerized (the v0.124.0 HIGH finding); `controller/scripts/docker_run_volume_path_gate.py` enforces (every `"-v"` allowlisted with its WHY).
- **Guarded file download (v0.124.0)**: `handler_export_download.go` — the canonical shape for streaming a server-side file to the browser: accept a BASENAME only (shape regexp + no separators/`..`), `filepath.Join` then assert `filepath.Dir(path) == dir`, `io.Copy` (never ReadAll), `Content-Disposition: attachment`, remove after a successful stream, TTL sweep (`sweepFabDownloads(dir, now, maxAge, logger)` — now injected for tests). Red-proof the guard by loosening to prefix-matching (the `..` case must fail).
- **Backups sub-page data**: `backupsCommonData(page, title, r)` + `backupsOffboxData(data)` (handlers.go) — the ONLY builders for the four `/backups*` pages; a new backups section extends these, never re-derives in a page handler. (The one-shot v0.124.0 move gate `backups_split_move_check.py` was retired in v0.126.0.)
- **App-list row (v0.126.0)**: `app_list_row`/`app_list_row_end` in `controller/internal/web/templates/app_row.html` is THE canonical list pattern — icon+name(+secondary) left, caller action block right; open with `dict "Slug" ... "Name" ...` (optional `Secondary`/`RowClass`/`Href`/`FallbackIcon`), close with `app_list_row_end`. Do NOT hand-roll app rows — `controller/scripts/app_row_dedup_gate.py` enforces single-sourcing (the backups_apps expander header is the one allowlisted aligned copy). Infra display identity: `inframeta.go` map + `infraMeta` func (filebrowser is the only Linked stack).
- **Consequential-action confirm (LIGHT)**: `felhomConfirm(el, question, onYes)` in layout.html (v0.123.0) — the trigger swaps in place to "kérdés + Igen/Mégse"; form buttons opt in with `data-confirm="…"` (delegated listener, `requestSubmit` keeps formaction/name-value). NEVER native `confirm()`/`prompt()` (OS-modals freeze browser automation — drill F-11; `native_confirm_gate.py` enforces). Heavy destructive flows keep the `.confirm-overlay``openDialog` pattern.
| timeAgo switch body ×2 | controller/internal/web/funcmap.go `timeAgo` vs `timeAgoStr` (identical formatting logic) |
| CSRF ×2 | controller/internal/web/csrf.go (session HMAC) vs controller/internal/setup/csrf.go (cookie double-submit) — intentional (pre-auth wizard) but unlabeled |
| Budapest timezone loader ×3 | controller/internal/scheduler/scheduler.go `getBudapestLocation` vs controller/internal/web/funcmap.go `getTimezone` vs controller/internal/quiesce/quiesce.go `budapestLocation` (v0.168.0 window gate — Budapest wall-clock, kept local to avoid a scheduler↔quiesce import edge) |
| JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` |
| DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` |
| compose exec ×2 | controller/internal/stacks/manager.go `composeExecCustomEnv` vs controller/internal/appexport/restore.go `composeExecEnv` (the latter has ctx+timeout; the former has the userdata belt) |
- Disk list sane: `curl -s https://felhom.demo-felhom.eu/api/disks` → felhom-usb (user-data, data_bearing), local/local-lvm (system), felhom-pbs (backup).
4. Record current deployed apps (so cleanup is unambiguous): dashboard "Alkalmazások", or `curl -s https://felhom.demo-felhom.eu/api/stacks/rescan` then the stacks list. (actualbudget is expected already deployed.)
- **Good:** all green/healthy; free space well above reserve. **Watch for:** any app stuck "Telepítés alatt" (deploying) from a prior run — note and resolve before starting.
Pick **two small apps** not currently deployed (suggest: `vikunja`, `mealie` — small images; avoid immich/nextcloud/paperless which are large pulls). UI: **Alkalmazások → <app> → Telepítés**, or API.
1. Deploy app #1 via UI (fill any required fields, generate passwords where prompted), click **Telepítés**.
- Watch the **3-step progress panel** (config → containers → health) and the live polling
(`GET /api/stacks/<app>` every 3s). API equivalent: `curl -s -X POST https://felhom.demo-felhom.eu/api/stacks/<app>/deploy -H 'Content-Type: application/json' -d '{"values":{}}'` then poll `GET /api/stacks/<app>`.
- **Good:** progresses config→containers→health; ends `running`/healthy within ~120s; the card flips to deployed; no "Telepítés" button reappears mid-pull (in-memory Deployed=true during pull).
- **Watch for:** stuck at a step, health-probe never going green (check the app's healthcheck tool exists), confusing Hungarian labels, the deploy gate returning **507** (insufficient Docker-data headroom — expected if low on space; note the banner wording).
2. Deploy app #2; same checks.
3. Confirm on disk the durable record is correct (CTRL-T2-1, happy case): `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks/<app>/app.yaml | grep deployed"` → `deployed: true` (only after success).
- **Good:** `deployed: true` on disk after a successful deploy. **Watch for:** secrets appearing in plaintext in app.yaml (they must be `enc:`-prefixed — H10/encryption check).
---
## 3. Deploy crash-window probe (THE CTRL-T2-1 test)
Goal: prove a crash during the image-pull window leaves the stack **NOT-deployed and redeployable**, not ghost-stuck.
1. Pick a **third app with a non-trivial image pull** (so the pull window is a few seconds — e.g. `paperless-ngx` if space allows, else `mealie`). Start the deploy (UI Telepítés or API POST), and **immediately** — while it is still pulling (status `deploying`, before `running`) — kill the controller:
-`ssh felhom-pve "pct exec 9201 -- docker kill felhom-controller"`**[operator: time this during the pull]**
- The bootstrap service (`felhom-controller-bootstrap.service`) restarts it within seconds. Confirm back up: `curl -s https://felhom.demo-felhom.eu/api/health`.
- UI/API: `GET /api/stacks/<app>` → state `not_deployed` (the card shows **Telepítés**, not a ghost "deployed").
3.**Redeploy** the same app — it must be **allowed** (no "already deployed; use update instead" refusal) and complete normally.
- **Good:** post-crash the app reads not-deployed and redeploys cleanly. **PRE-FIX behaviour (must NOT occur):** app.yaml `deployed: true` with no containers, and redeploy refused — that's the ghost-stuck regression the fix removes.
- **Watch for:** any orphaned containers from the killed pull (`docker ps -a`); the half-pulled image is fine (compose re-pulls).
1.**Export** a deployed app to a `.fab` bundle. UI: app page → **Exportálás** (or API `POST /api/export/start` with `{"stack_name":"<app>","drive":"<drive>"}`, poll `GET /api/export/status`). The `.fab` lands under the chosen drive's `exports/`.
- **Good:** export completes; a `.fab` appears; manifest readable via `POST /api/export/manifest`.
2.**Re-import** the same `.fab`: UI import flow (or `POST /api/export/import` with the file path), poll `GET /api/export/import/status`.
- **Good:** imports, recreates the stack, data restored; fail-closed data-key gate honored if the app has a data-encrypting key.
3.**Negative — path traversal (CTRL-001):** craft a hostile `.fab` and confirm it is **rejected at parse**, not written.
- Build a minimal bundle whose `manifest.json` has `"app_name":"../evil"` (and/or an `hdd_subdirs` / `volume_names` entry with `../`). Place it under a registered `exports/` dir on the host:
`ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller sh -c 'ls /mnt/felhom-usb/exports/'"` to find the dir.
- Attempt import of the hostile bundle.
- **Good (the fix):** import **fails immediately** with a manifest/validation error; **no directory is created outside the stacks dir** (verify: `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller ls -la /opt/docker/evil /etc/evil 2>/dev/null"` → nothing). **PRE-FIX (must NOT occur):** a dir/file written outside `/opt/docker/stacks/`.
- **Watch for:** the error message clarity (does the UI explain why it was rejected?).
- **Good:** completes; DB dumps written under `<drive>/backups/primary/<app>/db-dumps/`; recovery unit refreshed (`backups/primary/<app>/` has compose + manifest.json).
2.**Tier-2 off-drive copy:**`POST /api/backup/tier2`. (Single-drive demo → Tier-2 goes to the internal SSD for SMALL units only, DB/config-only, with the rootfs-headroom guard.)
- **Good:** the per-app "2. mentés" card in the UI populates (success → target drive, or an honest "needs 2nd HDD" no-target reason). **Watch for:** the rootfs-headroom guard correctly **refusing** rather than filling rootfs when the unit is too big for the SSD.
3.**Whole-guest (PBS) tier** is the agent's: confirm via the monitoring page that the PBS/offsite tier shows recent snapshots (the controller surfaces agent/PBS status; it does not run vzdump itself).
Re-confirm the two refusals proven on 2026-06-13. **Do NOT send a matching confirmation for a real data drive.** Target the real data-bearing device `/dev/sdb1` (felhom-usb) — both calls REFUSE, nothing is formatted.
1.**Refusal A — no confirmation:**`curl -s -w '\n%{http_code}\n' -X POST https://felhom.demo-felhom.eu/api/disks/format -H 'Content-Type: application/json' -d '{"device":"/dev/sdb1","fstype":"ext4"}'`
- **Good:** `formatted:false`, `needs_confirmation:true`, **HTTP 409**; no mkfs.
2.**Refusal B — wrong durable_id:** same call with `"confirmed":true,"durable_id":"byid:wwn-0xDEADBEEF-DOES-NOT-EXIST"`.
- **Good:** `formatted:false`, refused **409**; a non-matching confirmation does not authorize a wipe.
3.**Data-safety assertion:**`ssh felhom-pve "findmnt /mnt/felhom-usb -o TARGET,SOURCE,FSTYPE; pct exec 9201 -- docker exec felhom-controller sh -c 'df -h /mnt/felhom-usb'"` → still mounted, used space unchanged.
4.**Happy-path destructive wipe** = **[HUMAN]** — never wipe a real/customer drive to test; covered by the agent unit test `retarget-mismatch-refused`. Only on a genuinely disposable blank device, supervised. **[DESTRUCTIVE — operator confirm]**
Do this LAST, on a NON-critical app (e.g. one deployed in §2, not customer RomM data). Restore overwrites the app's current data.
1. Pick an app with a backup/snapshot from §5. Note its current data state (log in, note an item) so you can confirm the restore round-trip.
2.**[operator confirm]** Restore: `POST /backup/restore` (web form: `stack_name=<app>&snapshot_id=<id>`), or the dashboard restore UI.
- **Good:** the app stops, restores volume tars / recovery unit, restarts; data returns to the snapshot state; the fail-closed data-key gate refuses (with a clear message) if a data-encrypting key can't be recovered — in which case STOP and do NOT force.
- **Watch for:** data loss vs the snapshot, secret-regeneration (must recover from the guest's own app.yaml, regenerate nothing), confusing progress/labels.
3.**[operator]** Any **delete/remove** of a stack to clean up the §2/§3 test apps is **[DESTRUCTIVE — operator confirm]** — protected stacks (traefik/cloudflared/filebrowser/controller) must remain unstoppable server-side; confirm a delete attempt on one of those is refused.
---
## 8. Wrap-up
- Tear down the test apps deployed in §2/§3 (operator-confirmed deletes), leaving the demo as found (actualbudget + RomM intact).
- Write `REPORT-e2e-live-drive-<date>.md`: per-section ✓/✗, every UX/latency/error-handling finding, and any regression (especially: did §3 leave a ghost-deployed stack? did §4 write outside the stacks dir? did any wipe call format?). File new bugs against the audit record.
| R1 | python hog ~8 GB (~91% RAM), 75s | **PASS** | mid-stress: 10Gi used / 1.1Gi avail — controller **healthy**, 25/0 unhealthy; post: **dockerd NRestarts=0, no OOM** (alloc fit via cache eviction), recovered to 8.9Gi. Backstop pkill guard armed. |
| R2 | settings save on full disk | **PASS (code-verified)** | `settings.save()` = atomic write-`.tmp`-then-`os.Rename` (`settings.go:236-261`) → a disk-full fails at WriteFile, original untouched. Live disk-full not reproducible: settings.json is on the 252 GB docker vol. |
| R3 | fill rootfs (holds `/mnt/sys_drive` backups) to 94% (timed rm @150s) | **PASS** | controller stayed healthy (it's on the separate 252 GB vol); DB-dump + Tier-2 (8 apps, immich 155MB) **succeeded**; **settings.json stayed valid JSON** (no corruption). Reverted → rootfs 4%. (Literal-full refusal not pushed — near-0 rootfs risks wedging the guest OS unattended; the 2 GB margin had room so the gate didn't need to refuse.) |
| R4 | memory gate hard-block | **PASS (code-verified)** | `deploy.go:186` is a **hard block** (returns an error on `committed+new > usable`), using **committed-memory** accounting (sum of deployed `mem_request`), reserved 384 MB, usable 11904 MB. Live committed ≈ 4.4 GB → ~7.5 GB headroom, so tripping needs mass-deploying ~7.5 GB of requests (impractical/risky unattended); the branch + math are verified. |
## Phase 2 — State corruption [HIGH SEV] — **mostly PASS; 1 medium finding (S1)**
| # | Break (backed up first) | Result | Evidence |
|---|---|---|---|
| S1 | truncate settings.json → restart | **⚠ FINDING (medium)** | controller **crash-loops**: `[FATAL] Failed to load settings … unexpected end of JSON input`, RestartCount=7, restarting. **No safe-defaults fallback** — a corrupt settings.json takes the management plane down. *Not silent* (FATAL logged — better than the worst case). Restore → running/healthy, 3 storage_paths back. (Docker restart-manager cycling it re-confirms Finding #1.) |
| S2 | garbage in uptime-kuma `app.yaml` → rescan | **PASS** | controller stays **healthy** (25/0), logs `[WARN] LoadAppConfig: yaml: … did not find expected key` (not silent, not fatal), other apps unaffected. Restore → `running/deployed`. **Contrast with S1:** per-stack app.yaml corruption is graceful; settings.json is fatal. |
| S3 | corrupt quiesce marker (bad JSON) → restart | **PASS (minor finding)** | no panic/crash-loop, controller healthy, **stacks not stranded** (restarts=0). But the unparseable marker is **silently ignored** (no log) **and left in place** (not cleared/quarantined) — a real corrupted-mid-quiesce marker would skip recovery with no signal. |
| S4 | symlink `→ /etc` in userdata/appdata → backup | **PASS (security holds)** 🔒 | Tier-2 rsync **preserved the symlink** (`EVIL_ETC -> /etc`), did **not** follow it; **/etc contents did NOT leak** into the backup (no passwd/shadow). No path-escape/exfil. |
## Phase 3 — Concurrency storms — **PASS**
| # | Break | Result | Evidence |
|---|---|---|---|
| C1 | deploy + backup + restore + git-sync simultaneously | **PASS** | backup/run + tier2 both single-flighted ("Mentés már folyamatban"); restore mutex'd (302); sync ran independently ("nincs változás"); after settle **no stuck flag** (`running:false`), no deadlock, 25/0 healthy |
| C2 | rapid felhom-usb flap ×5 in ~10s | **PASS** | converged → felhom-usb **MOUNTED**, disconnected mark `None` (not stuck); **0 `permission denied`** during the flap; the v0.75 mountpoint-gate fired **5 clean skips** ("not mounted") in the disconnected windows; 25/0 |
## Phase 6 — Input/security fuzzing [HIGH SEV] — **PASS (no escape) + 1 medium defense-in-depth finding**
| # | Break | Result | Evidence |
|---|---|---|---|
| F1 | malformed/oversized/unknown-field JSON | **PASS** | not-json → **400**, empty → 400, 1MB → 400, nested-object-where → 400; unknown fields safely ignored (502 from a downstream non-existent mount, **no partial mutation**); controller **healthy** (no panic) |
| F2 | path traversal (storage `where`, restore `stack_name`) | **PASS (no escape) 🔒 + FINDING** | **storage:** all 5 traversals (`/etc`, `../../etc`, `/mnt/../etc`, `/mnt/felhom-drives/../../../etc`, `/etc/shadow`) **rejected** by `gateWhere` (`path.Clean`+`HasPrefix("/mnt/")`). **restore:** the boundary HELD — `/etc/passwd`**intact**, no `/opt/etc`/`/etc/passwd/` artifacts, no `/etc` writes — but **the restore handler does NOT reject a traversal `stack_name` upfront**: `RestoreFromRecoveryUnit("../../../etc")` proceeded (`GetAppDrivePath` → default `/mnt/sys_drive`), saved only by downstream **map-based**`StopStack`/`StartStack` (`stack "../../../etc" not found`) + no recovery-unit/volumes (no-op). **Defense-in-depth gap — recommend explicit `stack_name` validation at the restore handler.** (My own batched test looped these restores ×90s each, briefly holding the mutex — a test artifact that cleared when stopped, NOT a stuck-flag bug.) |
| F3 | hostile compose on a scratch app | **N/A (good property)** | the standard deploy renders only from the git-synced **catalog** — there is **no customer-facing arbitrary-compose injection vector**. (`.fab` import is the only path; not fuzzed — lower priority.) |
| F4 | weird userdata filenames (unicode, spaces, `(N)`, 200-char) | **PASS** | files created + Tier-2 backup ran clean ("Tier 2 run complete: 8 apps"), no crash/panic, 25/0. (`(N)` merge/dedup is migration-specific — `migrate.go` unit-tested, not live-run here.) |
| B1 | restart dockerd in 9201 | **PASS** | boot-restore brought all **25 containers + controller** back in <8s (drives stay mounted on a daemon-only restart → no boot-ordering issue); controller+cloudflared healthy |
| B3 | reboot 9201 with **felhom-flash detached** | **PASS** 🔑 | booted; flash **DETACHED** (disconnect intent persisted across reboot); 8 HDD apps **held** (absent, not crash-looping); **NO rootfs shadow dirs** (`…/felhom-flash/userdata` doesn't exist). Docker boot-restore made 2 bind-source mkdir attempts → **denied by the unprivileged-LXC mapping** → no escape (the v0.75 gate covers the controller's post-boot belt/FileBrowser; the unprivileged mapping covers docker's boot-restore). Reconnect → flash remounted, 8 apps recovered → 25/0. |
| B4 | host reboot of felhom-pve | **SKIPPED** | unattended risk — no physical recovery if the N100 doesn't POST/return. Deferred to supervised (consistent with #1/#2). |
_,_=w.Write([]byte(`{"ok":false,"error":"requested 3300 MB is too close to current usage 3000 MB (floor 3512 MB)","data":{"code":"below_usage_floor","usage_mb":3000,"floor_mb":3512,"min_mb":2048,"max_mb":14336}}`))
log.Printf("[WARN] [backup] ValidateDump: %s is structurally valid (%d tables) but its accounts table has NO rows — the dump may predate the customer's data",filePath,tableCount)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.