Testing: unit, integration, E2E and the FEAT-024 soak¶

Part of the temporary reversible deletion plan. Every slice is written test first. Anything that touches transactions, write conflicts or the fold runs against a real Mongo replica set (as ADR-020 requires); nothing runs against staging or production data. Heavy runs (the soak, the 100,000-Study benchmark) go to Bramble, not the shared CI host.

1. Unit tests (by slice)¶

Slice	What the tests pin
S1 Study marker and guard	The composite predicate (unlocked and unmarked); an unmarked Study serialises byte-identically to today; a marked Study's miss throws `StudyDeletedException`, mapped to HTTP 410 `study-deleted` and to the hub denial `StudyDeleted`; the guard's explanation distinguishes a lock from a marker
S2 Project state and gate	Deleted project: every `Project` and `Stage` policy answers 410 for a member and 404 for anyone else, in Off, Shadow and Enforced authority modes and on the SignalR handler; recovery routes are allowed for the owner (within the window) and application administrators only; each list query excludes deleted projects; the delegated-work gate denies a job for a deleted project; reservation timeouts and Bulk PDF clean-up still run; statistics maintenance skips a deleted project
S3a Operation store and runner	State machine is monotonic and has no erasing state; exact-token lease, renewal and takeover after expiry; one sweeping operation per project, further requests wait; `RestoreWindowEndsAtUtc` is snapshotted and never moves; the window check at read time; ledger records carry no free text (schema test)
S3b In-flight stage and upload cancel	The "belongs to the item" rule; the status-by-status table of in-flight work §2.2; cancellation command ids are idempotent; a cancel that loses to the agent's claim joins the wait set; the wait set for recalculations, exports and other sweeps; new imports, uploads and bulk updates refused at the route and at job start while a deletion is pending; the 24-hour blocked alert
S3c Import cancel	The status table of in-flight work §3.2; the `Error` claim with `CancelledForDeletion` is a no-op when repeated; the parse's start and per-batch checks stop it whatever the statistics flags; completion refuses a cancelled import
S3d Bulk-update cancel	The phase table of in-flight work §4.1; the durable cancel request is read by `StopReasonAsync`, the lock phase, the pre-lock phase and every takeover; the runner acquires only when no attempt is live; the search-scope test over planned Study ids; version 1 cancelled only before it applies
S4a Search delete	The fence declaration includes every project-wide family, QuestionAnswers and the search's SearchPopulation scope, keyed by the operation id, admitted only after the waits; the sweep filter excludes locked, already-marked and pending Studies; a re-run after a crash moves nothing twice; the link snapshot holds the import-job entry, and stage search filters are left unchanged
S4b Search restore	Returned Studies get their original project, lose the marker and gain a version; derived screening fields recalculated under the current settings; answers to deleted questions removed; link and import-job entry re-added; reservations never restored; refused after the window except for application administrators
S6 Project delete and restore	`IsProjectDeleted` written exactly when a statistics control exists; restore clears it without staling Fresh families; reservations recorded before the deletion are released on restore; refused after the window except for application administrators
S8 Administrator restore	The list includes in-window and after-window items; restore applies the same checks as a user restore; only the administrator capability may call either route; the ledger records the administrator
S5/S7 Web	Dialog copy and dates; "Deletion pending" with its reason; Recently deleted lists and their window; restore flows with typed refusals; SignalR events update the store; flag-off keeps the legacy immediate-delete success path (owner decision 2026-10-14) and the typed 409 refusals. Run with `pnpm exec ng test`, plus the repo-wide theme, zoneless and help-URL guard specs before every push

2. Integration tests (real Mongo replica set)¶

Leak catalogue. One fixture project with two searches, annotated and screened Studies, an allocation regime, a reconciliation stage and statistics enabled. Delete one search and assert that every read surface omits its Studies: the Library list and counts, screening and annotation allocation, reconciliation, review-eligibility pools, allocation reads, stage settings claim release, exports, the search list, SearchPopulation, every family's live calculation, and every id route (410 for the moved Study). The catalogue is a table the test iterates, so a new read surface is one row.
Write refusal. Every write path in ADR-020's table, against a marked Study, is refused with the typed StudyDeleted refusal (extend StudyBulkUpdateLockHonouringTests).
Forced interleaving. A save racing the sweep either lands before the move (kept, and back on restore) or is refused; it never lands after the move. Repeat for screening, annotation, reservation claim and the fold-path save.
Crash and takeover. Stop the runner during the wait, during the sweep and during a restore, then resume and check exact counts; restore from a partial deletion.
In-flight work (with RabbitMQ for the job paths).
Uploads: cancel then detach with an upload in each status; a cancel racing the agent claim; a delivery racing the move ends Failed without writing; clean-up of a cancelled upload completes after its search is unlinked and after its project is deleted.
Imports: cancel in each status; a batch saved just before the cancel is cleaned up; cancel versus completion in both orders; the saga ends Error; no unrevealed Study left; the reference file kept.
Version 2 bulk updates: cancel in each phase with a live attempt and with none; every applied change undone (before-image comparison); a cancel at Committed keeps the update; a quarantined rollback blocks the deletion.
Version 1 bulk updates: cancelled while validating, with no fence admitted and nothing applied; waited for while applying.
Late admissions: after the deletion's release, a redelivered parse start, a late failure replay and a stalled bulk-update attempt leave no active fence.
Crashes: the runner dies after each cancel and mid-rollback; the next lease holder finishes without cancelling twice.
Statistics, all nine families enabled, transactional path and fold path:
search delete → during the sweep, reads fall back; after release and rebuild, parity InParity for all families; QuestionAnswers counts drop by exactly the deleted answers;
search restore → InParity; drift check driftedStudies: 0;
fold mode with pending entries at deletion time → those Studies wait; zero pending sets under the holding id at the end;
project delete then restore with no activity → InParity immediately, before any backfill;
project delete, a reservation timeout and a late save while deleted, then restore → the changed families are Stale, every other family InParity at once, and all InParity after the backfill.
Restore after the project changed. Change the agreement threshold and delete a question while the search is deleted, then restore: drift check zero, answers removed, rebuilds succeed, parity exact.
Projects. Delete → every route and list refuses or omits; restore → statistics as in item 6.
The window. After RestoreWindowEndsAtUtc the item leaves the user lists and a user restore is refused; an administrator restore succeeds; nothing in the database changes when the window ends.
Nothing is erased. After delete, restore, window expiry and every background job, every document of the fixture still exists (count every collection before and after).
Restore-all (the rollback aid): restores every deleted item of an environment and leaves parity exact.

3. Architecture tests¶

Study writes. StudyWriteLockArchitectureTests requires the composite predicate; the deletion runner is allow-listed with its reason.
Id loads. Every Studies.Get* by id in the API and project-management either compares ProjectId or is allow-listed with a reason (new; same scanning approach).
No erasure. No production member deletes from pmStudy, pmSystematicSearch or pmProject outside an allow-list (the import's own compensation for an unrevealed import, and preview seeding), so a future change cannot add erasure by accident (new).

4. E2E specs (hermetic local stack)¶

Run with bash e2e/run-local.sh --spec <name>. The stack sets DeletionLifecycle=true on the API and project-management, a restore window of a few minutes (a local-only override that the production database name refuses) and a one-minute runner schedule. Users are the seeded admin, reviewer and standard accounts.

As built so far (S4a's PR 4). The stack applies the project-management migrations before its services start (e2e/scripts/pm-migrations.sh) and e2e/scripts/service-env.sh sets the flag and the one-minute schedule; the restore-window override is not set yet (S4b and S5 add it with the specs that need a window end). The S4a parts of two specs exist:

search-deletion-lifecycle.spec.ts: a project with a seeded search (5 Studies) and a second search imported through the UI (10); a second tab has the Library open. The admin deletes the seeded search through the Searches page dialog: the route answers 202 and the snackbar says "Deletion requested…"; the second tab receives SearchDeleted ("Search "E2E CSV Import" was deleted.") and its Library refreshes to "1 - 10 of 10" without a reload; the search leaves the list and the kept search counts 10. MongoDB: the operation is Deleted with StudiesMoved 5, every Study of the search has the holding id and the marker, no Study under the holding id has a pending fold set, the ledger has SearchDeletionRequested and SearchDeleted (count 5, the admin as actor), and the search document still exists (DL-R1S-01, DL-R1S-09, DL-R1S-10).
search-deletion-in-flight.spec.ts: a 100,000-reference import through the UI, deleted while it parses through the import-job DELETE route with the admin's session (the Searches table lists no row for a running import, only the operations summary does, so the page offers no delete action until the import ends; S5's UI decides whether it should): the route answers 202 Requested; the job ends Error with CancelledForDeletion and the operation id, no search document is revealed, the search ends Deleted with nothing moved and no Study of it left, the link snapshot keeps the cancelled job entry, and the ledger has ImportCancelledForDeletion (DL-R1W-16, search part). It takes about ten minutes: the cancelled parse's limit is seven. Parity is not asserted here: the ordinary stack runs with the materialized statistics flags off and no allowlisted project, so there is nothing to audit; the soak's searchDeleteDuringImport drill (S10) owns that part.

The rows below remain the target for the later slices.

Spec	Steps and assertions
`search-deletion-lifecycle.spec.ts`	Admin deletes a search → gone from the list and Library counts; Recently deleted shows "Can be restored until"; a reviewer with one of its Studies open gets the refusal on save; restore → decisions and annotations back; delete again, let the window pass → gone from Recently deleted; an application administrator restores it through the admin route
`search-deletion-in-flight.spec.ts`	Start a large import, delete the search while it parses: "Deletion pending", the import shows "Import cancelled: the search was deleted", no partial Studies appear, the cancelled import entry is in Recently deleted. Start a bulk study update that touches the search (version 2 stack) and delete the search: the update shows "cancelled, no changes applied", then the search is deleted
`project-deletion-in-flight.spec.ts`	Start a bulk study update and an import in a project, delete the project: both are cancelled, the bulk update's changes are not visible after a restore, and the restored project shows the cancelled jobs with their reason
`bulk-pdf-deletion.spec.ts` (Bulk PDF lane, `run:e2e-bulk-pdf`)	Start an upload, delete the search: the upload is cancelled, then the search is deleted; restore it and find PDFs delivered by an earlier upload still linked
`project-deletion-lifecycle.spec.ts`	Owner deletes → a second browser context (member) loses the project from the list and gets the deleted page on a direct link; owner restores → member sees it again; after the window the owner can no longer restore it and an administrator can
`deletion-permissions.spec.ts`	A reviewer cannot delete or restore; a project administrator can delete and restore a search; only the owner deletes the project; nobody sees a "delete permanently" action
`deletion-statistics-parity.spec.ts`	With the FEAT-024 flags on for the seeded project: delete and restore a search, then delete and restore the project; after each step run the backfills and the parity read for all nine families → `InParity`; after the project restore, parity before any backfill → `InParity`
UI acceptance	Screenshots of the dialogs and Recently deleted views in light and dark themes for preview acceptance (M3)

5. FEAT-024 soak additions (for the sibling PR on PR #4010's branch)¶

The soak driver lives in e2e/harness/statistics-soak/ on test/stats-isolated-soak-driver (PR #4010, head 73212e611b). These are the additions to make there; this plan does not write them.

Stack and setup.

Run with DeletionLifecycle=true on both hosts, a restore window of about 20 minutes and a one-minute runner schedule (local configuration only).
Each soak project gets a second, deletable search whose Studies receive normal traffic. Today POST /api/e2e-setup/project creates one search; extend it (test-only) with a SearchCount and return SearchIds and the Study ids per search, or import a generated file through the real import route. The extension is simpler and keeps setup fast.
The run record keeps, per project, its searches with their Study ids and states. Traffic that hits a Study of a deleted search, or a deleted project, gets 410 study-deleted or the project refusal; these count as designed refusals (like "sufficiently screened"), never as failures. Any 5xx stays a defect.
The soak has no Bulk PDF agent; upload cancellation is covered by the Bulk PDF E2E lane (§4).
The import drills use the real import route (a generated reference file uploaded to the local S3), so the saga, the parse job and their fences run as in production. The bulk-update drill runs with bulkStudyUpdateAtomicApply on (version 2); a variant with it off checks that a version 1 update that is already applying is waited for.

Traffic actions (low weights in TRAFFIC_MIX; deletion is a rare, bulk action):

Action	Weight	What it does
`searchDelete`	1	On a project whose deletable search is active: delete it, wait for `Deleted`, record the operation
`searchRestore`	1	On a project whose deletable search is deleted and in its window: restore it, wait for `Restored`

Delete and restore are counted as deletionOperations, separate from the 1,000 relevant mutations of the soak gate.

Drills (each ends with settledParity for all nine families, plus the deletion checks listed):

Drill	What it does	Extra checks
`searchDeleteUnderLoad`	A burst of screening, annotation and reservation traffic on the deletable search's Studies while it is deleted	No 5xx; refusals only after the move; zero pending sets under the holding id; no deletion-owned fence left
`searchDeleteDuringConfigChange`	Request a deletion while an agreement-threshold recalculation runs	The deletion waits (`Requested`) and completes after the recalculation; parity exact
`searchDeleteDuringImport`	Start a large import of a third search through the real import route, and delete that search while it parses	The import ends `Error` (`CancelledForDeletion`); no unrevealed Study left; no fence left by the import or the deletion; parity exact
`projectDeleteDuringBulkUpdate`	Start a version 2 bulk study update on a throwaway soak project and delete the project while it applies; restore it	The update is rolled back (the restored project's Studies match their before-images); parity exact after the restore
`searchDeleteBesideImport`	Start an import of another search and delete the deletable search	The other import completes untouched (recommendation: waited for); the deletion completes after it; parity exact
`searchRestoreAfterChange`	Delete; change the agreement threshold and delete a question; restore	Drift check `driftedStudies: 0`; QuestionAnswers parity exact
`pmRestartMidSweep`	Delete a large deletable search (several thousand Studies) and stop project-management mid-sweep; start it again	The operation resumes by lease takeover; moved count equals the search's Study count; nothing moved twice
`projectDeleteRestore`	Delete a non-pilot soak project; send traffic to it; restore	Reads refused while deleted; statistics reads report the project unavailable; parity before any backfill: `InParity` except the families the traffic changed, which are Stale; all `InParity` after the backfill
`restoreAfterWindow`	Delete the deletable search; let the window pass; restore through the administrator route	User restore refused after the window; administrator restore succeeds; parity exact
`nothingErased`	At the end of the run	Every Study, search and project created by the run still exists (counted in `mongo.mjs`)

Evidence. Per drill: operation ids (never names), Studies moved and skipped for pending entries, sweep duration, waits and their reasons, refusals by type, per-family parity, the drift check, fence diagnostics and the pending-under-holding count. The validator expects all nine families and treats a failed deletion drill as a failed run.

6. Performance checks¶

On Bramble with the FEAT-024 stats-large-v1 dataset at --scale 4 (100,000 Studies in one search): time the delete and restore sweeps, and measure screening-save p95 latency and Mongo CPU while a sweep runs. Initial target (accepted by Chris, 5 October 2026): a 100,000-Study search moves in under 10 minutes, and save p95 during the sweep stays within 20% of its idle value. The production-copy rehearsal (#4068) will show whether real searches come near that size (Q1).

7. Staging acceptance¶

On staging (seeded data; Chris and the occasional testers): delete and restore a search and a test project, including one with a PDF upload in progress; watch the six-hourly preflight parity for the pilot project through the same activity; confirm the user-guide pages match what the screens do and say that deleted data is kept. Nothing is tested on production.