Skip to content

ADR-024: Reversible deletion lifecycle for searches and projects

Status

In-Review, 2026-10-14. This ADR records the decisions Chris made on 5 October 2026 (plan #4070) and on 14 October 2026, when he asked to "fully implement the proper feature with the suggested soft delete implementation as planned".

The delivery plan, its MVP boundary, slices and acceptance criteria are in the temporary deletion lifecycle plan, and the delivery board tracks progress. This ADR is the permanent record of the design.

It supersedes two earlier designs, neither of which was ever merged:

  • the uncommitted August draft "ADR-014: reversible deletion and permanent tombstones", in .worktrees/bulk-pdf-deletion-lifecycle. That draft also clashed with the number of ADR-014, the PDF viewer;
  • PR #2934, the "durable lifecycle persistence kernel". Its unpushed commits are preserved on the branch archive/pr2934-unpushed-2026-10-14.

Both designs scheduled physical deletion after a 24-hour grace period and kept permanent tombstones. Both added a lifecycle object to every document and filtered every read. This ADR replaces both.

Context

  • Production deletes immediately and erases. Production runs the bulk-update baseline 104eca9. There, DELETE on a search deletes the search and its Studies, and DELETE on a project deletes the project, its searches and their Studies.
  • On main, the routes depend on the flag. #2927 made the routes answer a typed 503. Since #4172 (owner decision, 14 October 2026), deletionLifecycle off runs production's legacy immediate deletion, so main stays promotable. deletionLifecycle on answers the 503 until this lifecycle replaces it.
  • Studies are read by project. Almost every Study read is scoped by Study.ProjectId. The plan counts 109 repository calls, 35 shared filters and 20 files that open the collection directly (data model §2). pmStudy is about 10.7 GB in production.
  • There is already a write guard. ADR-020's study lock installed a registered IAggregateWriteGuard<Study>. Every Study write already goes through it, and an architecture test enforces that.
  • Statistics can already treat a project as deleted. FEAT-024 has a deleted-project switch, ProjectStatisticsControl.IsProjectDeleted, which nothing writes yet. It has operation fences, release as Stale, rebuilds and a per-family parity audit.
  • Index builds need a deployment step. ADR-023 moves index builds out of application startup and into reversible pre-deploy migrations.

Decision

1. Nothing is ever erased (D2)

No job, schedule, TTL, route or person physically deletes a deleted search's Studies, the search document, a deleted project, its statistics, its Bulk PDF records or any stored file. There is no "Delete permanently" action and no erasing state.

This keeps the integrated plan's rules ACD-R18, ACD-AE17 and AC-R2a-57 intact, and leaves the permanent-erasure policy T-POL-01 at "not approved", now for searches as well as projects. A future erasure policy needs its own decision, a legal review and its own ADR. This design keeps that possible: everything belonging to one deleted search sits under one holding id and one operation.

There are two exceptions:

  • A failed or cancelled import is still cleaned up. Its unrevealed Studies were never shown to anyone, so removing them is not deletion of user data (Q7).
  • While deletionLifecycle is off, the legacy immediate deletion erases, as production does today (#4172). Turning the flag off for a rollback therefore re-enables erasure for new deletions. Studies that a reversible deletion has already detached are never touched by it, because they carry a holding ProjectId.

2. Searches: detach and mark

A deleted search's Studies are moved out of the project:

  • their ProjectId is set to a holding id. The deletion operation mints it once, and no Project ever uses it;
  • each Study gets a marker, Study.Deletion, holding OperationId, OriginalProjectId and MarkedAtUtc;
  • Audit.Version moves, so a copy loaded before the move can never write over it;
  • the search is unlinked from the project. The operation keeps a snapshot of the import-job entry. Amended 2026-10-15 (as built in slice S4a): there is no list position to keep (SystematicSearchIds is a set), and stage search filters are neither changed nor snapshotted, because no read applies them today (data model §5.3).

Every project-scoped read stops seeing those Studies without being changed. Restore runs the same steps in reverse.

The alternatives were:

  • A: a marker plus a filter on every read;
  • B: a list of excluded searches on the project;
  • C: copying the Studies to a holding collection;
  • E: unlinking the search only.

They were rejected because B and A leak through any read that forgets the filter; C copies whole documents inside size-limited transactions; and E hides nothing (data model §3).

3. Projects: a state on the Project document

A deleted project gets a top-level Project.Deletion state, ready for canonical conversion (BC-R38). Its Studies are not touched. The central access gate does the rest:

  • every Project* and Stage* policy answers 410 to members and 404 to everyone else, in all three authority modes;
  • the SignalR handler refuses the same way;
  • every project list filters the project out.

Background jobs skip a deleted project, except reservation timeouts and Bulk PDF clean-up, which only release or clean up. Restore is a single write.

4. The Study write guard is a composite

The registered IAggregateWriteGuard<Study> is StudyWriteGuard. Its writable predicate is "not locked by a bulk update and not deletion-marked". This was delivered in slice S1a (#4175):

  • generic saves, direct writes and snapshot checks all go through it;
  • a refusal throws StudyDeletedException, which the API answers as HTTP 410 study-deleted, with no Retry-After, and the hub answers as AccessDenialReason.StudyDeleted;
  • the architecture test fails the build on a Study write that uses only the lock half.

Only the deletion runner may set or clear the marker, and it always moves Audit.Version when it does.

Reads that ignore ProjectId are audited separately (slice S1b): id loads must compare ProjectId, or treat a marked Study as not found. A no-erasure architecture test pins every remaining delete path to an allow-list with reasons.

5. The operation, its lease, the ledger and the runner

The operation. One pmDeletionOperation document per deletion is the durable intent, the progress record and the lock (S3a, #4190).

  • States only move forward. A search goes Requested → Deleting → Deleted → Restoring → Restored; a project goes Requested → Deleted → Restored. Deleted lasts until a restore.
  • The lease is exact-token. Only the holder renews or releases it, and only before it expires. A lease that has expired is taken over with a new token, and that takeover is how a crash at any point is resumed. Every write after the insert is a conditional $set, keyed on the token and the revision.
  • One sweep per project. A unique partial index on ActiveSweepProjectId allows only one sweeping deletion or restore per project. That field is present only while the operation is Deleting or Restoring.

The ledger. pmDeletionLedger is append-only and has no TTL. It holds no names, titles, reasons or other free text: the optional reason stays on the operation. Records have deterministic ids, so a retried append is the same record.

The runner. DeletionLifecycleRunner is host-neutral (ADR-021). A thin project-management consumer runs it on a recurring Quartz schedule (MissedEventPolicy.Send). It hands each operation to the stage handler for its scope and state:

  • the in-flight stage (S3b);
  • the sweeps (S4a, S4b);
  • the project steps (S6).

Its indexes are ADR-023 migrations only (0005–0008). They are never built at startup, and the runner advances nothing until they exist. No index on pmStudy is needed: the sweep walks the existing {ProjectId, _id} index under the holding id.

6. Running work is cancelled, then the deletion proceeds (D6, D12)

From the request on, new imports, uploads and bulk study updates in the project are refused. Then:

  • Cancelled: the item's own Bulk PDF uploads, through a shared cancellation service extracted from the cancel route; its search import, through an Error claim with CancelledForDeletion and the existing failure clean-up; and its version 2 bulk study updates, through a durable CancelRequest that rolls back. A bulk update is the item's when its plan touches the item's Studies.
  • Waited for: work past its point of no return; version 1 bulk updates that are already applying; and, as a recommendation, work in the same project that does not belong to the item.
  • Abandoned version 1 bulk updates (Chris, 14 October 2026, option A): the deletion waits for a version 1 bulk update at most 30 minutes since the job last changed. After that it treats the job as abandoned and proceeds. It logs a warning and appends one BulkStudyUpdateAbandonedForDeletion ledger record naming the job, so an engineer can see what was not waited for. Nothing about the job is changed or erased. This covers the held legacy jobs stuck in Parsing since September 2026.

No Study moves until every upload of the item is cancelled or finished. The Bulk PDF delivery probe refuses a whole batch if any Study in it is marked.

The deletion's statistics fence is admitted only after every cancelled or awaited job has released its own fence (in-flight work).

7. Statistics stay exact

Delete and restore are the only statistics events.

For a search. A search deletion or restore:

  1. admits one operation fence, keyed by the operation id (restores use their own namespace), over every project-wide family, QuestionAnswers (#3849) and the search's SearchPopulation scope;
  2. moves only Studies with no pending fold entries. The runner drains or discards the affected Studies' pending entries before marking, so none fold under the holding id;
  3. on restore, re-persists the derived screening fields and removes answers to questions deleted in the meantime, under its own fence;
  4. releases the fence as Stale, with a client invalidation. The rebuilds come from the scheduled stale repair (FEAT-024 C1) rather than from the runner: a per-family backfill can outlive the runner's lease, and fence-triggered rebuilds stay off for every fence owner until they get their own review. Stale rows are never served, so exactness does not depend on when the rebuild runs (as built in slice S4a-1).

For a project. Deletion sets IsProjectDeleted. The stored rows are kept and not served, and scheduled maintenance skips them. Restore serves them again, and every family that changed in between is already Stale.

A deleted project's statistics are frozen (Chris, 14 October 2026). Its kept statistics stay exactly as they were until it is restored. Nothing rebuilds, repairs, backfills or maintains them in between: the scheduled jobs skip it (slice S2b), and the explicit administrator maintenance and backfill routes refuse it, with the same refusal shape as the deleted-project access gate (slice S2a).

The per-family parity audit is the proof (statistics). The FEAT-024 owner reviews every slice that touches a fence.

8. The restore window and permissions (D1, D3)

The window. For 30 days (DeletionLifecycle:RestoreWindowDays), the item is listed in Recently deleted. The window end is snapshotted when the deletion completes and never moves. When it passes, nothing changes in the data.

Who can do what:

Action Who When
Delete or restore a search Holders of the RemoveSearch activity Restore within the window
Delete a project The owner only —
Restore a project The owner or an application administrator Within the window
Restore anything Application administrators Any time, through api/admin/deletions routes. There is no admin screen in the MVP

Application administrators are identified by the existing application-administrator capability (Chris, 14 October 2026, answering open question Q3): no new capability is named for deletion. The project recovery policy (slice S2a) and the administrator restore routes (slice S8) both use it.

A local and E2E minutes override exists for testing the window, and is refused on the production database.

9. Flag and rollout

Flag. deletionLifecycle is the existing deployment flag, on the API and project-management. It is not runtime-overridable. It gates creating deletions: endpoints refuse and the runner's schedule is not installed while it is off.

What stays on regardless. Guards and gates are not flag-gated: they do nothing until a marker or a deleted project exists, and they must keep refusing if the flag goes off. Already-persisted operations keep converging after the flag is turned off. Gaps in the on, off, on round trip are tracked in #4260.

Rollout. It follows ADR-020's pattern:

  1. Release 0: the guards ship dark everywhere.
  2. Release 1: the creators ship behind the flag.

An environment that turns the flag on must have run the pre-deploy migrations first. Production enablement is a separate step that Chris approves. main is promotable throughout, because flag off is production's behaviour.

Rolling images back to a build without the guards requires restoring every deleted item first (restore-all). An older build still reads every document, because Entity.ExtraElements keeps unknown fields, but it has no gate and no marker guard.

Consequences

  • Deletion becomes reversible and safe to promote. No read path changes. The leak surface is limited to reads that ignore ProjectId, and the id-load audit pins those.
  • Storage grows with every deletion and never shrinks (risk K11). Detached Studies keep their full documents and index entries, and gauges and a monthly review track the growth. A future erasure policy stays possible and scoped.
  • Statistics pay a fence and a rebuild for each search delete and restore. A project delete or restore costs no rebuild unless something changed while the project was deleted.
  • Large searches take minutes to sweep. The initial target is a 100,000-Study search in under 10 minutes, with save p95 within 20% of idle, measured on Bramble. During the sweep, the deletion shows as "Deleting…".
  • Turning the flag off brings erasure back for new deletions, because of #4172. Operators must weigh that before using the flag as a rollback lever where users can delete.

Alternatives considered

Alternative Why not
24-hour grace then physical deletion, with tombstones (August draft, #2934) Ruled out by D2. It also added state to every document and filtered every read
A purge after the window Ruled out by D2. After the window the item only leaves the users' view
A marker plus filters on every read (option A) One forgotten filter among more than 100 leaks deleted data
Copy to a holding collection (option C) Copies whole documents in size-limited transactions; id loads would need handling; restore copies back
Per-Study markers for project deletion The access gate hides a whole project with one write. Per-Study project markers stay with the integrated plan's canonical R0 and R2a work
Indexes built at startup on the new collections ADR-023: indexes are migrations. The new collections' indexes are migrations from the start

References