FEAT-024: Production-Copy Parity Rehearsal

This rehearsal restores a copy of the production SyRF database into the isolated local e2e stack, builds every materialised statistics family for every project, and judges exact parity family by family. It tests the real, older data shapes that synthetic soak traffic cannot produce. Chris approved it on 2026-10-05. It runs on Bramble only. Production itself is never touched: the copy is a local MongoDB container with no network route to Atlas.

Flag decision: none. This is local test tooling (e2e/): it changes no product code, no configuration of any deployed environment and no flag default. The --feat024-prodcopy mode of e2e/run-local.sh is opt-in.

The data and where it may be

An Atlas scheduled snapshot of Cluster0 is the whole production cluster. Besides the SyRF database (syrftest, about 10.8 GB on disk) it holds the other databases of the cluster (syrfdev, c3rs, email-logs and others) and the Atlas database users. It contains personal data: investigator names and email addresses, project names and membership, study titles, abstracts and authors, screening decisions, annotation answers and reviewer notes, and email logs.

Therefore:

• Download it yourself in the Atlas UI. Never share Atlas credentials or a download link with anyone, an agent included.
• Keep it on Bramble only, at ~/.syrf/backups/. Never copy it to Juniper (a shared, multi-user machine), never into a repository checkout (the tooling refuses), never to cloud storage.
• Delete it afterwards, together with the restored copy (step 6). The run keeps only pseudonymised evidence.

What the tooling guarantees

Guarantee	How
The copy is never named syrftest	The restore renames syrftest to syrf_prodcopy (or E2E_PRODCOPY_DB) and refuses syrftest in any case, syrf_e2e (dropped by every e2e start) and MongoDB's own names. The product's production refusals key on syrftest, so they stay meaningful.
Only the SyRF database is kept	Every other database of the snapshot is dropped, and so are the replica-set local database and the Atlas users and roles.
The restore can reach nothing	Each restore container runs with --network none; the snapshot is mounted read-only.
Nothing listens beyond this machine	Every container port is published on 127.0.0.1 only (e2e/docker-compose.prodcopy.yml) and the API and project-management bind to localhost. Quartz is not started.
Nothing leaves this machine	No seeding, no mail recipients (RestrictEmailToDev with no dev address), no Sentry or APM, and AWS clients get LocalStack and dummy credentials, never a profile from this host.
Only this stack's copy is used	The harness refuses any MongoDB but 127.0.0.1 on this stack's port, the configured non-production database, with the restore's marker present and no syrftest database on the server. run-local.sh refuses ConnectionStrings__MongoConnection__* settings in the environment that could point elsewhere.
The administrator is synthetic	The harness signs in through the E2E sign-in seam as a synthetic administrator (feat024-prodcopy-admin@syrf-rehearsal.invalid); the sign-in creates its Investigator inside the local copy only. No real account is used.
Evidence holds no personal data	Every evidence record is checked against an allowlist before it is written: pseudonymised project ids (p- and 12 hex characters of a keyed hash), counts, verdict codes, durations and metric key names. An unknown field, free text, an email address or anything that looks like an id is refused, and nothing is written.
Run-time files are private	Service logs, launchers, the allowlist and the private run state are created mode 0600 (directories 0700).

Before you start (Bramble)

• A worktree of this branch (or main once merged), built once by run-local.sh (or with --skip-build after a previous build).
• Disk (estimate, not measured on production data): the tarball itself, plus about three times the snapshot's unpacked size free on the Docker data root while the restore runs. A cross-database rename copies each collection before dropping the source, so the peak is the unpacked snapshot plus its largest collection. Allow 60 GB free; after the restore the copy settles at roughly the 10.8 GB of syrftest plus a 2 GB oplog and the statistics rows.
• Memory (estimate): the restore's mongod uses a 4 GB WiredTiger cache by default (E2E_PRODCOPY_WT_CACHE_GB; 8 to 16 on Bramble speeds the rename and the large projects). The running stack needs that cache plus about 1 to 2 GB each for the API and project-management and 3 to 4 GB for RabbitMQ, LocalStack, SQL Server and mock OIDC. Plan on 16 GB free.
• Time (estimate): extraction a few minutes; the rename 15 to 60 minutes for about 11 GB; stack start-up 2 to 5 minutes (longer once if the build adds indexes the copy does not have yet; the health wait allows 30 minutes). The rehearsal itself took 1 to 14 seconds per synthetic project of 8 to 32 studies (about 20 HTTP calls each); most production projects are small, and the largest are dominated by their backfills and parity audits. Calibrate with the --limit 20 pilot in step 3, then expect hours rather than minutes for every project.
• MongoDB versions. Bramble runs the stack on mongo:8.2 (E2E_MONGO_IMAGE=mongo:8.2; MongoDB 8.0.21 and later refuse its kernel version string; e2e/scripts/setup.sh applies the variable through the e2e/docker-compose.mongo-image.yml overlay when it is set, see the staging-proof runbook). A MongoDB 8.2 binary opens data at feature compatibility version 8.0 or 8.2 only. Note the cluster's MongoDB version in the Atlas UI when you download:
• 8.0 or 8.2: nothing to set; the stack image opens it.
• 7.0: the restore opens it with an 8.0 binary and raises the copy's compatibility version to 8.0. The restore image is pinned to mongo:8.0.20, the last 8.0 build before the kernel check, not the floating mongo:8.0, so nothing needs setting (mongo:8.0.20 is untested on Bramble; if it also refuses to start, set E2E_PRODCOPY_RESTORE_IMAGE to another build and report back). E2E_PRODCOPY_RESTORE_IMAGE always overrides the pin.

Step 1: download the snapshot (Atlas UI, yourself)

Atlas → Cluster0 → Backup → Snapshots → choose a recent scheduled snapshot → Download. Note the cluster's MongoDB version shown in the Atlas UI.

Step 2: put it on Bramble

mkdir -p ~/.syrf/backups && chmod 700 ~/.syrf ~/.syrf/backups
mv ~/Downloads/Cluster0-*.tar.gz ~/.syrf/backups/
chmod 600 ~/.syrf/backups/Cluster0-*.tar.gz

Step 3: run a pilot, then every project

From the worktree root on Bramble, a pilot of 20 projects first (it also performs the one-off restore):

E2E_MONGO_IMAGE=mongo:8.2 bash e2e/run-local.sh --feat024-prodcopy ~/.syrf/backups/Cluster0-<…>.tar.gz -- --limit 20

The first run checks the refusals, restores the snapshot into the Docker volume syrf-feat024-prodcopy (once; later runs reuse it), starts the stack on it, verifies that the stack's MongoDB serves the copy on loopback with no syrftest database, chooses the projects, starts the API and project-management with every statistics family on, and rehearses. For each project it:

1. records the statistics rows, publication guards and control the copy already holds, and the fold state (production may already hold some rows);
2. runs the read-only inclusion drift check and records drifted-Study counts by kind;
3. with --re-persist-drift only, re-persists the drifted Studies in the local copy and checks drift again (without it, families that read a drifted field refuse InclusionStatusDrift, which is itself a finding);
4. backfills each family in runbook order (ProjectScreening, MembershipScreening, ReviewerScreening, StageAnnotation, DomainReconciliation, MembershipStageAnnotation, ReviewerAnnotation, QuestionAnswers, SearchPopulation), retrying only the typed retryable refusals (FamilyFenced, RetryExhausted), and records duration, outcome and scope dispositions;
5. opens the project's narrow gate (the fleet gate is opened once per run);
6. takes exact per-family parity (GET …/parity?family=<family>) and records the verdict, the inconclusive reason, scope and metric counts and the names of any mismatched metric keys.

Every 5xx, timeout and refusal is recorded against its step. When the pilot looks right, run every project (the restore is reused):

E2E_MONGO_IMAGE=mongo:8.2 bash e2e/run-local.sh --feat024-prodcopy --skip-build -- --concurrency 3

Harness options (after --):

Option	Default	Meaning
--limit N	all	Rehearse the first N projects of the chosen order.
--projects id,id / --projects-file f	all	Only these projects (unknown ids are refused).
--order pseudonym\|studies-asc\|studies-desc	pseudonym	Pseudonym order mixes sizes stably.
--concurrency N	2	Projects in flight at once.
--batch-size N	3000	Projects per allowlist batch (at most 3,500: one 128 KiB environment string). The API and project-management restart with each batch's allowlist.
--re-persist-drift	off	Re-persist drifted Studies in the local copy before backfilling. Fixed for the run.
--request-timeout D	30m	Per-request timeout (90s, 15m, 1h). A timeout is recorded, not retried.
--stop-after N	none	Stop after N more projects (then --resume).

Environment: E2E_PRODCOPY_DB (default syrf_prodcopy), E2E_PRODCOPY_VOLUME (default syrf-feat024-prodcopy), E2E_PRODCOPY_RESTORE_IMAGE, E2E_PRODCOPY_WT_CACHE_GB (default 4), E2E_PRODCOPY_OPLOG_MB (default 2048), SYRF_PRODCOPY_OUT (default ~/.local/state/syrf/feat024-prodcopy).

Step 4: stop and resume

Ctrl-C once stops after the projects in flight and writes the evidence so far; a second Ctrl-C aborts. State is saved after every project. Continue with:

E2E_MONGO_IMAGE=mongo:8.2 bash e2e/run-local.sh --feat024-prodcopy --resume --skip-build

A resumed run continues at the first unfinished batch; a project interrupted mid-way, or one that hit network errors (the stack went away), is rehearsed again, keeping its first record of pre-existing state. Three projects in a row with network errors stop the run. The containers and the restored copy stay up between runs (loopback only) until step 6.

Step 5: read the evidence

Under ~/.local/state/syrf/feat024-prodcopy/run-<time>/:

Path	Contents
evidence/summary.md	The human summary: per-family verdict counts, inconclusive reasons, mismatched metric keys, backfill outcomes, backfill and parity p50/p95/max, projects with failures, the largest projects by studies, drift counts, the statistics state found in the copy, narrow-gate outcomes, errors and refusals.
evidence/summary.json, evidence/projects.jsonl, evidence/run.json	The same, machine-readable: one allowlisted record per project, plus run metadata (commit, restore versions and counts, options, progress). Safe to share: pseudonyms, counts, codes, durations and metric key names only.
private/state.json, private/run.log	Real project ids, for resuming. Stays on Bramble; deleted by cleanup --purge-private.
pseudonym.key (one level up, shared by every run)	The key behind the pseudonyms. Stays on Bramble; deleted by cleanup --purge-private, after which the evidence cannot be linked back to project ids.

Verdict categories, per family and project: InParity (exact), NotInParity (Mismatch: at least one served value differs from the authoritative calculation), Inconclusive (NotServed, SourceAdvanced or ScopesTruncated), NothingToCompare (NoScopes with nothing materialised and no scope state on either side, the soak's isEmptyParity rule: for example QuestionAnswers before any annotation answer) and Error (an HTTP failure, or an answer for another family). A project counts as failed for any NotInParity or Error, or any step that did not succeed (a refused or failed backfill, drift check, re-persist or narrow gate). Inconclusive is listed separately: it is not a pass.

To look at a failing project yourself, map its pseudonym back on Bramble only, before you purge (it needs the key and the run's private/state.json, both deleted by --purge-private):

node -e 'const { createHmac } = require("crypto"); const fs = require("fs");
const root = process.env.HOME + "/.local/state/syrf/feat024-prodcopy";
const key = fs.readFileSync(root + "/pseudonym.key", "utf8").trim();
const state = JSON.parse(fs.readFileSync(process.argv[1], "utf8"));
for (const id of Object.keys(state.projects))
  if ("p-" + createHmac("sha256", key).update(id.toLowerCase()).digest("hex").slice(0, 12) === process.argv[2]) console.log(id);' \
  ~/.local/state/syrf/feat024-prodcopy/run-<time>/private/state.json p-<pseudonym>

Step 6: clean up

bash e2e/run-local.sh --feat024-prodcopy-cleanup --purge-private
command rm -i ~/.syrf/backups/Cluster0-<…>.tar.gz
docker volume ls | grep prodcopy   # expect nothing

The cleanup stops the stack that served the copy, removes the restored data volume, deletes the service logs, launchers and allowlist it left in /tmp, deletes every run's private state (with --purge-private) and prints the exact command to delete the snapshot. The pseudonymised evidence is kept. bash e2e/scripts/prodcopy.sh status shows what is still present.

--purge-private also deletes the pseudonym key (~/.local/state/syrf/feat024-prodcopy/pseudonym.key). Anyone holding that key and a list of project ids could re-compute the pseudonyms and re-link the kept evidence to projects, so after a purge the evidence can no longer be linked to project ids, by you or anyone else. The consequence: a later run makes a new key, so pseudonyms are not stable across runs after a purge (the same project gets a different p-… in the next run's evidence), and the mapping snippet in step 5 no longer works for the purged runs. Without --purge-private the key and each run's private/ directory stay on Bramble, and the evidence can still be mapped back there.

How the restore works

e2e/harness/statistics-prodcopy/restore.mjs, called by run-local.sh:

1. Preflight. Refuses a production or reserved database name, a snapshot that is not a .tar.gz or .tgz, a snapshot inside any checkout of this repository, and Mongo connection settings in the environment. Reuses the volume when it already holds a complete copy of the same snapshot file (name and size); refuses a different snapshot or an interrupted restore until you clean up.
2. Extract. Unpacks the tarball into the volume and finds the directory holding the WiredTiger file (Atlas nests it one level down). Reads the MongoDB version from restoreInfo.txt when the snapshot carries one.
3. Choose the binary. The stack image if it opens the snapshot's version; otherwise the one release in between (MongoDB 8.0 for a 7.0 snapshot under an 8.2 stack), pinned to mongo:8.0.20 because 8.0.21 and later refuse Bramble's kernel version string (SERVER-125742). With no stated version it tries the stack image, then mongo:8.0.20. E2E_PRODCOPY_RESTORE_IMAGE overrides.
4. Prepare. Starts that binary standalone on the volume and renames every collection of syrftest to the target database (recreating views), drops every other database, the replica-set local database and the Atlas users and roles, raises the feature compatibility version only when the stack's binary could not open it otherwise, refuses if a syrftest database remains, and writes a marker (versions and counts) into feat024_prodcopy_meta.restore.
5. Shut down cleanly and write a completion file into the volume.

The stack then starts MongoDB on the volume as replica set rs0 (its health check initiates it), and restore.mjs verify-stack checks the mount, the loopback-only port, the marker and the absence of syrftest before anything else connects.

Troubleshooting

Symptom	Cause and remedy
The restore mongod (…) could not start or open the snapshot (a version, compatibility or kernel-version message)	The restore image refused to start on this host, or the snapshot's MongoDB release needs another binary. The message names E2E_PRODCOPY_RESTORE_IMAGE: set it (see "MongoDB versions" above), run the cleanup, start again.
Volume … holds an interrupted restore	A restore was stopped part-way. Run the cleanup, start again.
Refusing this MongoDB / Refusing this stack	The harness or the stack check found something other than this stack's restored copy. Read the listed reasons; do not work around them.
Services not healthy	tail -50 /tmp/<stack>-api.log (the stack name is in bash e2e/scripts/prodcopy.sh status). The first start on a copy may build indexes.
Many InclusionStatusDrift refusals	Expected on older data: rerun with --re-persist-drift (local copy only) to see the families' parity after the remedy the statistics reference prescribes.

Testing the tooling without production data

The tooling was built and tested on Juniper with synthetic data only:

• node --test e2e/harness/statistics-prodcopy/lib.test.mjs: the pure rules (refusals, MongoDB version path and the pinned restore image, pseudonyms, the evidence allowlist, resume state, verdict roll-up).
• bash e2e/scripts/prodcopy.test.sh: the cleanup, against a stub docker and a synthetic output root (--purge-private deletes every private/ and the pseudonym key, and keeps the evidence).
• bash e2e/harness/statistics-prodcopy/synthetic-snapshot.sh --out <dir outside the repo> builds an Atlas-like snapshot: a MongoDB 7.0 replica set with its own set name, a syrftest database filled by project-management's own seed-data mode (five generated projects), other databases, an Atlas-style user and role and a view, shut down cleanly and tarred as one directory with a restoreInfo.txt. Run the rehearsal on it exactly as in step 3, then clean up.