Skip to content

Reversible deletion lifecycle: delivery plan

Temporary planning package. Delete it when the programme is complete; the permanent record will be the deletion ADR (ADR-024) and the feature documentation. Nothing here changes product code, enables a flag or deletes data.

What was decided (Chris, 5 October 2026).

  • "Build reversible deletion now", for searches and projects, tested against the FEAT-024 materialised statistics.
  • "But no actual permanent deletion." Nothing is ever physically erased: no study, search, project or stored file is deleted, by a job or by a person. The integrated plan's rule stands: permanent physical erasure is the unapproved policy T-POL-01.
  • A 30-day restore window. For 30 days a deleted search or project appears in Recently deleted and authorised users can restore it. After that it leaves the users' view and only an application administrator can restore it. The data stays in the database, detached and marked, indefinitely.
  • Running work is cancelled, then the deletion proceeds: Bulk PDF uploads (D6, the 12 August draft's approach), search imports and bulk study updates (D12). A failed or cancelled import still cleans up the Studies it wrote before it was ever shown (Q7).
  • Permissions, decisions D4, D5 and D7 to D11, and the answers to the open questions are recorded in decisions, risks and open questions.

The plan on one page

What users get (the MVP). Deleting a systematic search or a whole project works again and can be undone. A deleted search appears under Recently deleted on the project's Searches page; a deleted project appears under Recently deleted projects on the projects page for its owner. Each entry says who deleted it, "Deleted on " and "Can be restored until ". Within those 30 days, the people allowed to delete it can Restore it, and everything comes back: references, screening decisions, annotations and saved sessions. After 30 days the entry leaves the list; an application administrator can still restore it through an administrator route. While something is deleted it is hidden from screening, annotation, allocation, exports and statistics; a deleted project is hidden from all of its members. Work in progress on the item is cancelled first (PDF uploads, imports, bulk study updates, which are rolled back); work past its point of no return is waited for; the deletion then completes on its own.

How it is stored (recommended model: "detach and mark"). A deleted search's Studies are moved out of the project by changing their ProjectId to a holding identifier owned by the deletion, and each one carries a small deletion marker that remembers where it came from. The search is unlinked from the project, with the link kept in the deletion record. A deleted project only gets a state on its own document; its Studies are not touched. Almost every Study read in SyRF is already scoped by ProjectId, so detaching hides the Studies from more than a hundred read paths without changing any of them, and the existing ADR-020 write guard (enforced by an architecture test) refuses writes to marked Studies. Detached data simply stays where it is. Evidence: data model.

Statistics stay exact. Delete and restore are the only statistics events, because nothing is ever erased. Deleting a search fences every family it changes, moves the Studies, releases the families as Stale and rebuilds them, so its Studies are recounted out; restoring recounts them in the same way. A deleted project's stored statistics are kept, not served (through ProjectStatisticsControl.IsProjectDeleted, which the readers already check but nothing writes today) and served again on restore. Each step ends with the per-family parity audit. Details: statistics.

Shortest critical path. The deletion operation store → the in-flight stage and the three cancel paths (uploads, imports, bulk updates, built in parallel; imports and bulk updates after #4062) → search delete → search restore → search UI, with the Study write guard built alongside. Alongside: project state and access gate → project delete and restore → project UI. The administrator restore route is small and follows both. See slices.

Contents

Part What it covers
MVP boundary (below) What is in the first usable release and what is not
Behaviour What users see, who can do what, Recently deleted, the restore window, messages, audit
In-flight work D6 and D12: cancelling Bulk PDF uploads, search imports and bulk study updates, then deleting; what is waited for
Data model The options with code evidence, the recommendation, the shapes and sequences
Restore window and kept data The 30-day window, administrator restore afterwards, what is kept, storage cost, the erasure non-goal
Statistics Per family: what delete and restore do; fences, rebuilds, parity and drift checks
Testing Unit, integration, architecture and E2E tests; the FEAT-024 soak additions
Rollout Flags, migration from the fail-closed state, staging and production, the production baseline branch, rollback
Acceptance criteria Numbered, testable criteria with their verification method, per release
Slices PR-sized slices in critical-path order, parallel work, large or risky slices
Decisions, risks and open questions What is decided, the remaining recommendations, risks and open questions

Starting point, verified

Each fact from the brief was checked against the code on main (at 35407997e), the production baseline commit and the local cluster-gitops checkout. Nothing was read from any live database.

# Fact Status Evidence
F1 Production runs API 9.44.1 built from the bulk-update production baseline; search DELETE there calls RemoveAndDeleteSystematicSearchFromProjectAsync and deletes the search and its Studies at once Confirmed. Production API and project-management run image tag …-restore.104eca9, built from 104eca9b05. The branch codex/bulk-update-production-baseline has since moved to ba8c8130e5 (the #3425 merge), which contains 104eca9b05 cluster-gitops/syrf/environments/production/{api,project-management}/config.yaml; git show 104eca9b05:…/SearchController.cs lines 74-93
F2 On the baseline, project deletion is also immediate Added. The brief mentions search deletion only. On the baseline, DELETE /api/projects/{id} calls DeleteProjectAsync, which deletes the project, its searches and all their Studies. So the next production release from main removes both search and project deletion git show 104eca9b05:…/ProjectController.cs lines 277-286
F3 On main and staging, #2927 makes project DELETE and both search DELETE routes throw DeletionLifecycleUnavailableException (typed 503) for both flag values Superseded on 2026-10-14 (owner decision, syrf#4172): with deletionLifecycle off the routes perform production's legacy immediate deletion again; with it on (staging) they keep the 503 until the lifecycle lands, except that both search routes record a reversible deletion since S4a (rollout §2). Originally confirmed. Feature commit 16b6e2efbc, merged on 28 August as 830e265b34. Staging runs main at 535ad97 (API 9.194.0) with deletionLifecycle: true and bulkPdfUpload: true on the API SearchController.cs:109-136; ProjectController.cs:401-421; staging api/values.yaml:152-156
F4 Staging project-management has the deletion flag Corrected. Only the staging API values set deletionLifecycle; project-management takes the chart default (false). The flag is declared for both services env-mapping.yaml:786-794; staging project-management/values.yaml
F5 The project-management deletion methods still exist and nothing outside tests calls them Superseded on 2026-10-14 (syrf#4172): they are the flag-off branch again, called by the three routes when deletionLifecycle is off. Originally confirmed, and DeleteProjectAsync was also uncalled. RemoveAndDeleteSystematicSearchFromProjectAsync checks the ADR-020 bulk-update guard and admits a FEAT-024 bulk fence ProjectManagementService.cs:1503-1540 and 1558-1660
F6 The fence declaration "is ready" and "belongs with the scheduler" Partly true. The declared set omits QuestionAnswers (deleting annotated Studies changes its counts; tracked in #3849), and it keys the fence on the search id, which is not unique once a search can be deleted, restored and deleted again ProjectManagementService.cs:109-137; ProjectScreeningInclusionRecalculation.cs:119-173; statistics-reference.md QuestionAnswers "Known gaps"
F7 #3082: the orphaned methods save or delete outside their transaction Partly outdated. The search method now saves inside its session; DeleteProjectAsync still deletes Studies after the commit. The issue's own action stands: do not resurrect as written #3082; ProjectManagementService.cs:1521-1539
F8 The integrated review plan reserves reversible project deletion as X-DEL / OS-A25 Confirmed and consistent. It also says "no job, TTL or scheduler physically deletes a deleted project's data" (ACD-R18, ACD-AE17, AC-R2a-57; T-POL-01). Chris's "no actual permanent deletion" keeps that intact, and extends it to searches specifications/spec-access-communications-and-deletion.md §3.3, §5, §11; acceptance-criteria.md:529
F9 (Not in the brief) An uncommitted deletion design exists Added. .worktrees/bulk-pdf-deletion-lifecycle (branch feat/bulk-pdf-deletion-lifecycle, based on August main) holds an untracked ADR-014-reversible-deletion-and-permanent-tombstones.md with product decisions approved on 12 August (24-hour grace then physical deletion, Project Admin schedules, upload fencing) and about 79 modified plus many untracked source files. ADR-014 on main is the PDF viewer; ADR-022 is the next free number That worktree's git status; source-status-inventory.md item 17
F10 (Not in the brief) Statistics already model a deleted project Added. ProjectStatisticsControl.IsProjectDeleted and ProjectDeletedAtUtc are read by the serving gate, the bundle reader, the fold applier, backfills, repair and seven query paths, but nothing writes them ProjectStatisticsControl.cs:159-164; ProjectStatisticsFoldApplier.cs:138-146
F11 (Not in the brief) Only Bulk PDF has a user cancel path Added. POST …/searches/{searchId}/bulkPdfUpload/uploads/{uploadId}/cancel accepts Queued, Uploading and unclaimed Uploaded jobs and refuses once the agent has claimed one. Search imports and bulk study updates have no cancel route. The version 2 bulk-update executor already rolls back on a cancelled token; version 1 (what staging and production run, since bulkStudyUpdateAtomicApply is unset) has no rollback BulkPdfUploadController.cs:536-582; BulkPdfUploadJob.cs:819-839; BulkStudyUpdateExecutor.cs:286-330

MVP boundary

The MVP is the smallest release that restores deletion for users, makes it reversible and keeps every statistics family exact. It is Release 1 in the rollout, preceded by a behaviour-neutral Release 0 that installs the guards.

In the MVP

  1. Search deletion, reversible. Delete a search (and the provisional import-job variant). Its Studies leave the project; the search appears in Recently deleted for 30 days.
  2. Search restore within the 30 days by RemoveSearch holders. Everything returns exactly, except temporary review places (reservations), which are released and never revived. Derived screening fields are recalculated against the current settings, and answers to questions deleted in the meantime are removed.
  3. Project deletion, reversible. The owner deletes a project; it disappears for every member and from every list; the owner sees it in Recently deleted projects for 30 days.
  4. Project restore within the 30 days by the owner or an application administrator. The project returns with its members, searches and data; its kept statistics are served again.
  5. After the window, restore by application administrators only, through an administrator API route (recommendation: no admin screen in the MVP; see restore window).
  6. In-flight work. The item's Bulk PDF uploads (D6), search imports and bulk study updates (D12) are cancelled, then the deletion proceeds; bulk updates are rolled back. Work past its point of no return, and (recommendation) work in the project that does not belong to the item, is waited for.
  7. Exact statistics across delete and restore for all nine families, proved by per-family parity in tests, in E2E and in the soak drills.
  8. Audit records. An append-only, value-free ledger of who deleted and restored what and when; an optional reason is kept on the deletion record.
  9. Tests and documentation. Unit, real-Mongo integration, architecture, E2E and soak additions; the user guide pages that describe deletion (user-guide/projects/settings.md, user-guide/studies/nav-studies.md, user-guide/studies/manage-studies.md), which must say plainly that SyRF keeps deleted data; the engineering docs.

Not in the MVP

Item Status
Permanent erasure of studies, searches, projects or stored files, by any job or person Non-goal. T-POL-01 stays "not approved". A future erasure policy needs its own decision, legal review and ADR (restore window §6)
An administrator screen for after-window restore Recommendation: the admin API route is enough for the MVP; a screen later if support needs it
A user-facing "Cancel" button for imports and bulk study updates The MVP builds the cancel paths for deletion only; exposing them to users is a later choice
Fencing upload processing that is already under way (the August draft's final-share write permits) Not needed: processing past the cancel point is waited for
Per-Study deletion markers for project deletion and the composite write guard over canonical collections (AC-R2a-53) Integrated plan: R0 composite guard, R2a (claims and drafts), R3a (pools)
Active-work impact preview with digest, required reason, C20 history events Integrated plan, X-DEL canonical parts
Emails and inbox notices about deletion Notification programme (G-NOTIF, OS-A26)
Search withdrawal (P1: hides Studies, keeps Citations) Integrated plan P1, with the F-P "deletion-lifecycle design join"
Withdrawing a pending deletion request Later, if users need it; restore covers mistakes

How this fits the integrated review plan (X-DEL)

The integrated review plan reserves reversible project deletion as X-DEL (OS-A25, T-AC-03, ACD §3.3, §4.5, §4.6, decision D3-12), "before R6", with canonical parts in R2a and R3a. This plan pulls X-DEL forward for today's legacy data model, extends it to searches, and leaves the canonical parts where the integrated plan put them.

X-DEL element (ACD reference) This plan Stays deferred to
Hide from lists; block review, editing and allocation (ACD-R14, AC-R2a-52) MVP, through the project access gate and list filters —
Stop project notifications (ACD-R14) Nothing is sent today; the gate stops SignalR traffic G-NOTIF capture rule
Release active reservations; never revive them (ACD-R14, ACD-R17, AC-R2a-54, AC-R2a-55) MVP: released by the existing timeouts, plus a restore-time release of any reservation older than the deletion Claims and drafts in R2a
Keep drafts, data and history (ACD-R14) MVP: nothing is changed while deleted, and nothing is ever erased —
Restricted Deleted projects view; restore (ACD-R17, B4) MVP: Recently deleted projects for the owner during the 30-day window; application administrators restore through an admin route at any time. The ACD lets owners see deleted projects indefinitely; Chris's window narrows that A restricted admin screen
Records with actor, time, reason, preview digest (ACD-R15, AC-R2a-56) MVP: actor, time, optional reason, counts Preview digest with R2c's impact preview; C20 events
Stale saves never bypass deletion (ACD-R16, AC-R2a-53) MVP for searches (marker plus write guard); for projects the access gate R0 composite guard and per-Study project markers
No scheduler physically deletes a deleted project (ACD-R18, ACD-AE17, AC-R2a-57) Consistent: nothing is ever erased, for projects or searches —
Search withdrawal is distinct from deletion (ACD-R19, AC-P1-16r) Kept distinct; MVP search deletion is today's deletion made reversible and never erases P1
Deleted projects convert in their deleted state (BC-R38) The deletion state is a top-level Project field, ready for conversion R6 waves
The deletion ADR, renumbered (T-AC-03 blocker) ADR-024 replaces the uncommitted ADR-014 draft (slice S0; ADR-022 and ADR-023 were taken meanwhile) —