D5: an app restore works from the drive alone (v0.188.0)
The recovery unit on the customer's drive now carries the PORTABLE secret class, so Tier-1/Tier-2 restore no longer depends on the whole-guest tier. A customer needs the drive and nothing else. Part 0's rulings overturned the brief's recommendation, on evidence: - the data_key flag is untrustworthy (4+ encryption keys the catalog itself labels as such are unflagged) -> R-127 - a DB password is not resettable in practice: POSTGRES_PASSWORD is ignored once PGDATA is non-empty, so a regenerated value leaves the app unable to authenticate against its own restored rows while the dump replay still reports success (proven on a throwaway postgres:16-alpine) Ruling (operator): type:secret travels, type:password never does, minus the nonPortableSecrets code register. Plaintext -- withholding the internet- reachable class is what licenses that, and the two are coupled. Precedence: the UNIT WINS over the guest -- the unit's secrets were captured in the same run as the dumps beside them, so they match the data being restored. The fail-closed data-key gate is unchanged. Secret values are never logged; the manifest records NAMES only.
This commit is contained in:
@@ -15,20 +15,28 @@ import (
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// RecoveryManifest describes an app's self-contained, SECRET-FREE recovery unit (Phase 2).
|
||||
// RecoveryManifest describes an app's self-contained recovery unit.
|
||||
//
|
||||
// The unit on a drive is `<nsRoot>/backups/primary/<app>/` and contains:
|
||||
//
|
||||
// compose/ docker-compose.yml + .felhom.yml + a SECRET-STRIPPED app.yaml
|
||||
// compose/ docker-compose.yml + .felhom.yml + app.yaml (0600; carries the PORTABLE secrets)
|
||||
// db-dumps/ app-consistent DB dump(s) (written by the dump flow)
|
||||
// volume-dumps/ named-volume tars (written by the dump flow)
|
||||
// manifest.json this file
|
||||
//
|
||||
// The unit holds NO secret values, NO data-encrypting keys, and NOT the Docker image — only the
|
||||
// pinned image tag(s) (re-pulled on restore) and the NAMES of the secret/data-key env vars. The
|
||||
// secret values are recovered at restore time from the guest's own app.yaml (live on the rootfs,
|
||||
// or via the PBS whole-guest snapshot) — see Restore. "Restore from the unit alone" is therefore
|
||||
// honestly "unit + the guest's app.yaml"; SecretSource records that dependency explicitly.
|
||||
// D5 (schema 2) changed what the unit holds. Before it held NO secret at all, which made
|
||||
// "restore from the drive alone" false: the fast, local, customer-doable Tier-1/2 restore secretly
|
||||
// depended on the slow, operator-driven whole-guest restore, because a data-encrypting key or a DB
|
||||
// password absent from the guest cannot be regenerated without rendering the restored data
|
||||
// unreachable. The unit now carries the PORTABLE secret class (stacks.PortableSecretEnvVars) in its
|
||||
// 0600 app.yaml, and Tier-1/2 needs the DRIVE AND NOTHING ELSE.
|
||||
//
|
||||
// It still holds NO `type: password` admin login (those are internet-reachable, so their blast radius
|
||||
// is not bounded by the drive — they stay in the guest and are regenerated on restore) and NOT the
|
||||
// Docker image, only the pinned tag(s), re-pulled on restore. SecretSource records the split.
|
||||
//
|
||||
// A schema-1 unit carries no secrets: the restore degrades to the pre-D5 guest-only behaviour rather
|
||||
// than failing, and the next capture rewrites it (the app.yaml checksum changes).
|
||||
type RecoveryManifest struct {
|
||||
SchemaVersion int `json:"schema_version"`
|
||||
AppName string `json:"app_name"`
|
||||
@@ -38,13 +46,17 @@ type RecoveryManifest struct {
|
||||
Drive string `json:"drive"` // HDD_PATH (in-guest mount)
|
||||
NamespaceRoot string `json:"namespace_root"` // resolved felhom-data namespace root
|
||||
ImagePins []string `json:"image_pins"` // image NOT stored — re-pulled on restore
|
||||
SecretEnvVars []string `json:"secret_env_vars"` // NAMES only — recovered from guest/PBS
|
||||
SecretEnvVars []string `json:"secret_env_vars"` // NAMES of every secret/password field
|
||||
DataKeyEnvVars []string `json:"data_key_env_vars"` // fail-closed gate on restore
|
||||
SecretSource string `json:"secret_source"` // human note: where secrets come from
|
||||
ConfigFiles []string `json:"config_files"` // captured into compose/
|
||||
DBDumps []string `json:"db_dumps"`
|
||||
VolumeDumps []string `json:"volume_dumps"`
|
||||
Checksums map[string]string `json:"checksums"` // sha256 of captured compose/ files
|
||||
// PortableSecretEnvVars (D5) are the NAMES of the secrets this unit's app.yaml CARRIES. Names only
|
||||
// — the manifest is 0644 and never holds a value. The restore reads it to know which app.yaml env
|
||||
// entries are secrets rather than plain config; absent (schema 1) ⇒ the unit carries none.
|
||||
PortableSecretEnvVars []string `json:"portable_secret_env_vars,omitempty"`
|
||||
// R-43/R-44 (v0.148.0): the coherence stamp. An offsite run refreshes the dumps FIRST and then
|
||||
// captures the unit, so a manifest carrying an OffsiteRunID asserts "the db-dumps/ in this unit
|
||||
// were taken by that run" — i.e. the snapshot is an internally coherent {DB@T, files@T} pair.
|
||||
@@ -68,9 +80,11 @@ func (m *Manager) SetTier2Notifier(fn func(stackName, destLabel string, dur time
|
||||
m.tier2Notify = fn
|
||||
}
|
||||
|
||||
// CaptureRecoveryUnit writes/refreshes an app's secret-free recovery unit: it captures the
|
||||
// compose + metadata + a secret-stripped app.yaml into compose/, enumerates the DB/volume dumps
|
||||
// already present, and writes manifest.json. It NEVER writes a secret value or the Docker image.
|
||||
// CaptureRecoveryUnit writes/refreshes an app's recovery unit: it captures the compose + metadata +
|
||||
// an app.yaml carrying the PORTABLE secret class (D5) into compose/, enumerates the DB/volume dumps
|
||||
// already present, and writes manifest.json. It never writes the Docker image (only the pinned tag),
|
||||
// and never writes a WITHHELD secret — the split is decided in buildUnitAppYaml, pinned by
|
||||
// TestCaptureRecoveryUnitCarriesPortableSecretsOnly.
|
||||
//
|
||||
// Idempotent: it builds the captured content in memory first and SKIPS all writes when the unit is
|
||||
// already current (same config checksums, same dump set, same controller version) — so it can run on
|
||||
@@ -107,7 +121,7 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
checksums[fname] = sha256Hex(data)
|
||||
configFiles = append(configFiles, fname)
|
||||
}
|
||||
appYaml := buildStrippedAppYaml(info)
|
||||
appYaml := buildUnitAppYaml(info)
|
||||
files = append(files, capFile{"app.yaml", appYaml, 0600})
|
||||
checksums["app.yaml"] = sha256Hex(appYaml)
|
||||
configFiles = append(configFiles, "app.yaml")
|
||||
@@ -151,30 +165,34 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
}
|
||||
|
||||
manifest := &RecoveryManifest{
|
||||
SchemaVersion: 1,
|
||||
AppName: stackName,
|
||||
DisplayName: info.DisplayName,
|
||||
ControllerVer: version,
|
||||
CreatedAt: time.Now().UTC().Format(time.RFC3339),
|
||||
Drive: drivePath,
|
||||
NamespaceRoot: nsRoot,
|
||||
ImagePins: info.ImagePins,
|
||||
SecretEnvVars: info.SecretEnvVars,
|
||||
DataKeyEnvVars: info.DataKeyEnvVars,
|
||||
SecretSource: "guest app.yaml (live rootfs) or PBS whole-guest snapshot — never stored in this unit",
|
||||
ConfigFiles: configFiles,
|
||||
DBDumps: dbDumps,
|
||||
VolumeDumps: volDumps,
|
||||
Checksums: checksums,
|
||||
OffsiteRunID: runID,
|
||||
DumpsAt: dumpsAt,
|
||||
SchemaVersion: 2, // D5: compose/app.yaml carries the portable secret class
|
||||
AppName: stackName,
|
||||
DisplayName: info.DisplayName,
|
||||
ControllerVer: version,
|
||||
CreatedAt: time.Now().UTC().Format(time.RFC3339),
|
||||
Drive: drivePath,
|
||||
NamespaceRoot: nsRoot,
|
||||
ImagePins: info.ImagePins,
|
||||
SecretEnvVars: info.SecretEnvVars,
|
||||
DataKeyEnvVars: info.DataKeyEnvVars,
|
||||
PortableSecretEnvVars: info.PortableSecretEnvVars,
|
||||
SecretSource: "portable secrets (data keys, DB passwords, internal signing secrets) are IN this unit's compose/app.yaml (0600); internet-reachable admin logins are NOT, and come from the guest's app.yaml or are regenerated on restore",
|
||||
ConfigFiles: configFiles,
|
||||
DBDumps: dbDumps,
|
||||
VolumeDumps: volDumps,
|
||||
Checksums: checksums,
|
||||
OffsiteRunID: runID,
|
||||
DumpsAt: dumpsAt,
|
||||
}
|
||||
if err := writeManifest(manifestPath, manifest); err != nil {
|
||||
return fmt.Errorf("writing manifest: %w", err)
|
||||
}
|
||||
|
||||
m.logger.Printf("[INFO] [backup] Recovery unit captured for %s → %s (images=%d, secrets-referenced=%d, data_keys=%d)",
|
||||
stackName, RecoveryUnitPath(nsRoot, stackName), len(info.ImagePins), len(info.SecretEnvVars), len(info.DataKeyEnvVars))
|
||||
// Counts and NAMES only — never a value (D5 puts more secrets through this path than before).
|
||||
m.logger.Printf("[INFO] [backup] Recovery unit captured for %s → %s (images=%d, secrets-referenced=%d, data_keys=%d, portable-carried=%d/%d, withheld=%d)",
|
||||
stackName, RecoveryUnitPath(nsRoot, stackName), len(info.ImagePins), len(info.SecretEnvVars),
|
||||
len(info.DataKeyEnvVars), len(info.PortableSecrets), len(info.PortableSecretEnvVars),
|
||||
len(withheldSecretNames(info)))
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -201,29 +219,63 @@ func (m *Manager) versionLocked() string {
|
||||
return m.version
|
||||
}
|
||||
|
||||
// strippedAppYaml is the on-disk shape of the secret-free app.yaml captured into the unit.
|
||||
// strippedAppYaml is the on-disk shape of the app.yaml captured into the unit. The name is historical:
|
||||
// since D5 the `env` map carries the PORTABLE secrets alongside the plain config (see buildUnitAppYaml).
|
||||
type strippedAppYaml struct {
|
||||
Deployed bool `yaml:"deployed"`
|
||||
Env map[string]string `yaml:"env"`
|
||||
}
|
||||
|
||||
// buildStrippedAppYaml renders a secret-free app.yaml (non-secret env only) as bytes. Deterministic:
|
||||
// yaml.v3 sorts map keys and the secret-name list comes in stable metadata order, so identical input
|
||||
// yields identical bytes (needed for the checksum-skip guard).
|
||||
func buildStrippedAppYaml(info RecoveryInfo) []byte {
|
||||
body, err := yaml.Marshal(strippedAppYaml{Deployed: true, Env: info.NonSecretEnv})
|
||||
// buildUnitAppYaml renders the unit's app.yaml as bytes: the non-secret env PLUS the portable secret
|
||||
// values (D5). Deterministic: yaml.v3 sorts map keys and the name lists come in stable metadata order,
|
||||
// so identical input yields identical bytes (needed for the checksum-skip guard).
|
||||
//
|
||||
// This is the ONE place the capture side decides what does and does not reach the drive — there is no
|
||||
// second path that writes a unit app.yaml. The caller writes the result 0600.
|
||||
func buildUnitAppYaml(info RecoveryInfo) []byte {
|
||||
env := make(map[string]string, len(info.NonSecretEnv)+len(info.PortableSecrets))
|
||||
for k, v := range info.NonSecretEnv {
|
||||
env[k] = v
|
||||
}
|
||||
// Portable secrets last: NonSecretEnv is disjoint from the secret set by construction
|
||||
// (GetStackRecoveryInfo), so this cannot shadow a plain config value.
|
||||
for k, v := range info.PortableSecrets {
|
||||
env[k] = v
|
||||
}
|
||||
body, err := yaml.Marshal(strippedAppYaml{Deployed: true, Env: env})
|
||||
if err != nil {
|
||||
body = []byte("deployed: true\nenv: {}\n")
|
||||
}
|
||||
header := "# Captured by felhom-controller recovery unit — SECRET-FREE.\n" +
|
||||
"# Secret/data-key values are intentionally omitted; recover them at restore from the\n" +
|
||||
"# guest's own app.yaml (live rootfs, or the PBS whole-guest snapshot). Stripped names:\n"
|
||||
if len(info.SecretEnvVars) > 0 {
|
||||
header += "# " + strings.Join(info.SecretEnvVars, ", ") + "\n"
|
||||
header := "# Captured by felhom-controller recovery unit.\n" +
|
||||
"# This file CARRIES SECRETS (D5) so a Tier-1/2 restore needs the drive and nothing else:\n" +
|
||||
"# data-encrypting keys, database passwords and internal signing secrets. Mode 0600.\n"
|
||||
if len(info.PortableSecretEnvVars) > 0 {
|
||||
header += "# Carried: " + strings.Join(info.PortableSecretEnvVars, ", ") + "\n"
|
||||
}
|
||||
// The withheld class is named, not valued — an operator reading the unit must be able to see WHY a
|
||||
// credential is missing rather than suspecting a capture bug.
|
||||
if withheld := withheldSecretNames(info); len(withheld) > 0 {
|
||||
header += "# WITHHELD (internet-reachable logins — stay in the guest, regenerated on restore): " +
|
||||
strings.Join(withheld, ", ") + "\n"
|
||||
}
|
||||
return []byte(header + string(body))
|
||||
}
|
||||
|
||||
// withheldSecretNames returns the secret names deliberately NOT carried by the unit, in stable order.
|
||||
func withheldSecretNames(info RecoveryInfo) []string {
|
||||
portable := make(map[string]bool, len(info.PortableSecretEnvVars))
|
||||
for _, n := range info.PortableSecretEnvVars {
|
||||
portable[n] = true
|
||||
}
|
||||
var out []string
|
||||
for _, n := range info.SecretEnvVars {
|
||||
if !portable[n] {
|
||||
out = append(out, n)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// writeManifest writes the manifest JSON atomically.
|
||||
func writeManifest(dst string, manifest *RecoveryManifest) error {
|
||||
data, err := json.MarshalIndent(manifest, "", " ")
|
||||
|
||||
Reference in New Issue
Block a user