RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite 540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest, booted, verified and destroyed itself; zero 990000 guests or volumes afterwards and both local-lvm figures returned to their pre-run values. Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for the first time: never-proven sorted first, each box took its HOST tier, deferred the offsite one, and picked it up on the next evaluation six hours later. The host-tier proofs reached the hub through R-189's merge — demo-felhom's report carries two tiers, and the local one can only have come from disk. The capability map's optimistic half is cashed, with its scope stated: these two boxes, not the fleet. Surfaced and filed rather than fixed: - R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with a reinstall, any logged pveum activity and any cluster-log entry ruled out. - R-191: every weekly offsite backup uploads successfully and then fails the job on a prune the box is deliberately not allowed to do (R-89 moved it server-side; both boxes still arm keep_last=2). Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the write path (six, with the hub raising whole_guest_backup_failed at the first), and my earlier "no restore_test_* events" was produced by grepping a 404 page.
This commit is contained in:
@@ -1,137 +1,173 @@
|
||||
# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)
|
||||
# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
|
||||
|
||||
> **A LATER SESSION THE SAME DAY (R-189 · R-188 · R-186) touched this repo for REGISTERS AND DOCS
|
||||
> ONLY** — `OPEN-ITEMS.md`, `STATUS.md`, `CONTEXT.md` (S-19, S-20) and the capability map. **No hub
|
||||
> code changed and the hub was not bumped**: the defect was the agent no longer sending
|
||||
> `restore_tests[]`, not the hub failing to read them. That session's report is
|
||||
> `felhom-agent/REPORT.md`. What follows is the last hub IMPLEMENTATION, v0.91.1, which is what this
|
||||
> file is for.
|
||||
**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
|
||||
and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
|
||||
box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
|
||||
`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
|
||||
source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
|
||||
|
||||
**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1**
|
||||
(`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not
|
||||
after it. Ruling recorded: **ep0 is Tier 2, protected**.
|
||||
**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
|
||||
|
||||
## 1. Baselines, re-read on arrival
|
||||
---
|
||||
|
||||
| Repo | `main` @ commit | Version | Matched §1? |
|
||||
|---|---|---|---|
|
||||
| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** |
|
||||
## 1. Preconditions
|
||||
|
||||
The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in
|
||||
`REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and
|
||||
`REPORT.md` is overwritten every session, so the per-repo history under-reported what was running.
|
||||
| # | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
|
||||
| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
|
||||
| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
|
||||
| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
|
||||
| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
|
||||
| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
|
||||
| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
|
||||
|
||||
## 2. Why Part 2 could not ship later
|
||||
|
||||
The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved
|
||||
weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment
|
||||
derived that number *from the cadence R-86 removes*.
|
||||
|
||||
The brief said a weekly tier would "sit exactly on that line". That is literally true, and the
|
||||
arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the
|
||||
current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false
|
||||
by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late,
|
||||
one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
|
||||
|
||||
## 3. The window as derived
|
||||
## 2. The due verdicts before the run, quoted
|
||||
|
||||
```
|
||||
interval = declaredArchiveInterval(tier) // host 26h, offsite 8d
|
||||
if observedOK && observed > interval { interval = observed } // observation may only WIDEN
|
||||
window = clamp(4 * interval, floor 7d, cap 12d)
|
||||
demo-felhom tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst"
|
||||
reason: newest settled archive … has not been proven; nothing proven on this tier yet
|
||||
tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z"
|
||||
reason: … has not been proven (last proven archive was a different one)
|
||||
|
||||
demo-hp tier=felhom-backup due=true archive="…2026_08_02-04_49_29.tar.zst"
|
||||
tier=felhom-pbs due=true archive="…2026-07-28T19:19:45Z"
|
||||
```
|
||||
|
||||
- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same
|
||||
tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience.
|
||||
- **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very
|
||||
thresholds the backup-freshness checker already judges those tiers against. Not a second opinion:
|
||||
if the two checkers disagreed about how often a tier receives an archive, one would be alarming on
|
||||
the other's model.
|
||||
- **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful
|
||||
`backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both*
|
||||
arrays, and classifying by array membership would attribute an offsite archive to the host tier).
|
||||
Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
|
||||
- **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before.
|
||||
- **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is
|
||||
never called stale against an archive PBS has already pruned.
|
||||
**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
|
||||
|
||||
| tier | interval used | window | proof age in health | verdict |
|
||||
## 3. The runs — all four SCHEDULED, none triggered
|
||||
|
||||
| box | tier | due at | archive | result |
|
||||
|---|---|---|---|---|
|
||||
| daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before |
|
||||
| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin |
|
||||
| newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes |
|
||||
| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
|
||||
| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
|
||||
| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
|
||||
| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
|
||||
|
||||
## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box
|
||||
**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
|
||||
only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
|
||||
which start nothing.
|
||||
|
||||
demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and
|
||||
`2026-07-28T04:49:43Z` — **8 h 54 m apart**, because one is a healing artefact and the other a real
|
||||
weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor
|
||||
lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm
|
||||
this whole task exists to prevent would have returned within a week, on the box it had just shipped
|
||||
to.**
|
||||
**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
|
||||
proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
|
||||
evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
|
||||
is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
|
||||
|
||||
Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means
|
||||
nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real
|
||||
information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared
|
||||
rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a
|
||||
signal meaning *unverified* — *broken now* is `restore_test_failed`, immediate and untouched.
|
||||
**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The
|
||||
tier an ordinary recovery uses is also the cheapest to prove.
|
||||
|
||||
## 5. Kept, because it was earned
|
||||
## 4. What the runs left behind
|
||||
|
||||
- **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged).
|
||||
- **The stale signal stays edge-triggered** (`staleStates` untouched).
|
||||
- **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken,
|
||||
the other that it is unverified, and the second is the one that quietly becomes the first.
|
||||
- **Every reason string now states the window it was judged against** — R-100's corollary: when a
|
||||
verdict changes what it counts from, the alarm text must change with it, or an operator reads
|
||||
"limit 168h" under a tier judged at 288 h and dismisses a true alarm.
|
||||
- **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window
|
||||
and to see two generations of a weekly tier.
|
||||
**Persisted state — v3, naming the archive, the tier and what was verified:**
|
||||
|
||||
## 6. Tests and red-proofs
|
||||
```json
|
||||
demo-felhom felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
|
||||
proven_at 2026-08-03T22:56:45Z}
|
||||
felhom-pbs → {archive …2026-07-28T04:49:43Z, tier pbs, proven_at 2026-08-04T05:04:21Z}
|
||||
demo-hp felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
|
||||
felhom-pbs → {archive …2026-07-28T19:19:45Z, tier pbs, proven_at 2026-08-04T06:10:39Z}
|
||||
```
|
||||
|
||||
| Test | Asserts | Mutation | Observed |
|
||||
|---|---|---|---|
|
||||
| `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL** — `week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` |
|
||||
| `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval` → `observed > 0` | **FAIL** — `window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` |
|
||||
| `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass |
|
||||
| `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass |
|
||||
| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged |
|
||||
**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
|
||||
latest report holds **two** entries, one per tier; the `local` one can only have come from the
|
||||
persisted state, because the in-memory store held only that morning's offsite run:
|
||||
|
||||
**A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular
|
||||
weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on
|
||||
exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while
|
||||
looking like proof. The jitter is what makes it a test — and it is also the truth about the old
|
||||
constant.
|
||||
```
|
||||
demo-felhom tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
|
||||
tier=pbs pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z tested_at=2026-08-04T05:04:21Z
|
||||
demo-hp tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
|
||||
```
|
||||
|
||||
## 7. Deployment
|
||||
**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
|
||||
|
||||
| Step | Evidence |
|
||||
|---|---|
|
||||
| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` |
|
||||
| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` |
|
||||
| ArgoCD deliberate sync | `sync=Synced health=Healthy` |
|
||||
| Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` |
|
||||
| Startup log | checkers initialized; `Listening on :8080` |
|
||||
```
|
||||
tier=felhom-pbs due=false proven="…2026-07-28T04:49:43Z"
|
||||
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
|
||||
```
|
||||
|
||||
## 8. Part 3 — the ep0 ruling, recorded
|
||||
…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
|
||||
That is not a defect: it is "proved daily, on its own archive", visible one day later.
|
||||
|
||||
`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"*
|
||||
for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table
|
||||
names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**.
|
||||
`CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex,
|
||||
Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores,
|
||||
prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the
|
||||
ordinary off-site read a restore-test performs remains permitted** — which this session then exercised.
|
||||
## 5. Teardown — three layers, per box
|
||||
|
||||
## 9. Registers and docs
|
||||
| layer | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| the machine | `pct list` → **0** entries for 990000 | **0** |
|
||||
| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
|
||||
| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
|
||||
|
||||
- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row.
|
||||
- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific.
|
||||
- **R-185 / R-186 / R-187 → filed** (grep established 185–187 free; R-184 was the highest in use).
|
||||
- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself
|
||||
proposed** — which was the never-fires version.
|
||||
- `STATUS.md` rewritten for the operator, trimmed back to one screen.
|
||||
- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test
|
||||
row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key)
|
||||
and S-18 (ep0).
|
||||
Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
|
||||
|
||||
## 6. What the run surfaced — three findings, two of them corrections to my own record
|
||||
|
||||
### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
|
||||
|
||||
A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
|
||||
2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
|
||||
~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
|
||||
(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry.
|
||||
Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before.
|
||||
**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
|
||||
and v0.123.0's probe detects the *state* but says nothing about the *transition*.
|
||||
|
||||
### R-191 (new) — every weekly offsite backup reports FAILED although it worked
|
||||
|
||||
demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
|
||||
incrementally) and the job then failed on the prune —
|
||||
`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
|
||||
errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
|
||||
(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
|
||||
the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
|
||||
must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
|
||||
— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
|
||||
(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
|
||||
namespaces.
|
||||
|
||||
### Two corrections to yesterday's record
|
||||
|
||||
1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
|
||||
**six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
|
||||
`whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
|
||||
said backups kept landing because writes go through a root path — wrong, and now corrected in the
|
||||
runbook and on the row.
|
||||
2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
|
||||
I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
|
||||
since the R-86 work — and one **more** since, below.
|
||||
|
||||
**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
|
||||
2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
|
||||
host tier had never been proven, and the hub said so on its own, without being asked. It has not
|
||||
re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
|
||||
|
||||
## 7. The capability map
|
||||
|
||||
The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
|
||||
boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
|
||||
teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
|
||||
is untested and untouched, and one box proving something does not make it a fleet property.
|
||||
|
||||
## 8. Registers
|
||||
|
||||
- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
|
||||
correction in §6.
|
||||
- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
|
||||
the highest in use).
|
||||
- `ROADMAP.md` holds none of these rows, so nothing to collapse.
|
||||
- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
|
||||
"What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
|
||||
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
|
||||
returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
|
||||
warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
|
||||
the restore pointed at `/mnt/nvme-1tb` instead.
|
||||
- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
|
||||
none". Nothing depends on that assumption now, but the operations notes still carry it.
|
||||
- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
|
||||
role — a 2.7× difference worth understanding before either is used to size anything.
|
||||
- **Both waiters this session produced no output** despite the runs completing; the evidence was
|
||||
gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
|
||||
instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
|
||||
the hub's database, not on the waiters.
|
||||
|
||||
Reference in New Issue
Block a user