Skip to content

Delivery board: reversible deletion lifecycle

Part of the temporary reversible deletion plan. This board tracks delivery of the slices. The slices, their contents and the criteria are unchanged except where §5 says so.

Mandate (Chris, 14 October 2026): "fully implement the proper feature with the suggested soft delete implementation as planned". Everything stays behind the existing deletionLifecycle flag, default off.

Flag semantics since syrf#4172 (owner decision, 14 October 2026). With the flag off (the default, and production's value), the three DELETE routes run production's legacy immediate deletion. With it on (staging), they answer the typed 503 until S4a and S6 replace that with the lifecycle. Every slice below changes only the flag-on branch, or adds guards that do nothing until a deletion exists. Staging runs the flag on, so staging exercises each slice as soon as it is wired.

Decisions on 14 October 2026 (Chris), recorded in ADR-024 §6, §7 and §8 and in decisions and risks:

  • Q3, answered. A deleted project is restored by its owner within the restore window, or by an application administrator at any time, through the existing application-administrator capability. No new capability is needed. Before S6 or S8 wires the recovery policy to a route, fold in the S2 follow-ups in #4204, starting with registering the recovery handler scoped rather than as a singleton.
  • D13. The explicit FEAT-024 administrator maintenance and backfill routes refuse a deleted project, so its kept statistics stay frozen until it is restored. This is slice S2c, which reuses the S2a refusal shape and is tested with a positive control.
  • S3d, version 1 bulk updates stuck in Parsing: option A. The deletion waits at most 30 minutes since the job last changed, then treats it as abandoned and proceeds. It logs a warning and appends one BulkStudyUpdateAbandonedForDeletion ledger record naming the job; nothing is erased. Implemented in S3d-2 (#4213), criterion DL-R1W-17.

Promotion rules for every slice ("main always promotable"):

  • Persisted changes are additive and rollback-safe. The production build (104eca9b05) must read anything a slice writes, through Entity.ExtraElements. Untouched documents stay byte-identical.
  • No slice builds an index on pmStudy (about 10.7 GB in production) at start-up. A slice that needs one declares it as a pending pre-deploy migration under ADR-023 (#4177, #4179, #4180). The two new small collections (pmDeletionOperation, pmDeletionLedger) are empty when they are created, so they may build their indexes at start-up.
  • Server-gated: every endpoint, consumer and job that creates or advances a deletion checks deletionLifecycle. Guards (the Study write guard and the project access gate) are not gated on the flag: they do nothing until a marker or a deleted project exists, and they must keep refusing writes if the flag is later turned off.

1. Order of PRs (critical path first)

# Slice PR Depends on State
1 S1a Study marker and composite write guard (dark) #4175 — Merged (14 October)
2 S3a Deletion operation store, ledger and runner (dark) #4190 — Merged (14 October)
3 S0 Deletion ADR (ADR-024) #4191 — Merged (14 October)
4 S2 Project deletion state and access gate (dark). Split (§5 item 15): S2a state, gate, recovery policy, list filters and the delegated-work denial; S2b the other background-job rules S2a #4194; S2b #4195 S2b on S2a Both merged (14 October)
4c S2c FEAT-024 administrator maintenance, backfill, rebuild, re-persist and mode routes refuse a deleted project (decision D13): 410 project-deleted, statistics frozen until restore #4211 S2a Merged (15 October)
5 S1b Study id-load audit and no-erasure architecture tests #4189 #4172 merged Merged (14 October)
6a S3 seam: the Requested stage, in-flight participants, the pending-deletion guard (409 deletion-pending), migration 0008 #4203 S3a Merged (15 October)
6 S3b Upload cancel and the other waits (an in-flight participant). Split: S3b-1 the upload cancel path, participant and guards; S3b-2 the waits for recalculations, exports and other deletions, and the clean-up tests S3b-1 #4206; S3b-2 #4212 S3 seam S3b-1 and S3b-2 merged (15 October)
7 S3c Import cancel (an in-flight participant), split in two: S3c-1 the cancel path, S3c-2 the participant and the new-work guards S3c-1 #4205; S3c-2 #4214 S3 seam (#4062 merged on 5 October) S3c-1 and S3c-2 merged (15 October)
8 S3d Bulk-update cancel (an in-flight participant) #4207 (S3d-1: the durable cancel in the executor); #4213 (S3d-2: the participant and the guards) S3 seam (#4062 merged) S3d-1 merged (15 October); S3d-2 open, under review
9 S4a Search delete (flag on). Split: S4a-1 the statistics fence; S4a-2 the runner's hand-off, unlink, sweep and completion; S4a-3 the routes create the deletion (202), client and minimal web; S4a-4 the SignalR SearchDeleted event, the E2E spec and the benchmark S4a-1 #4238; S4a-2 #4240 (merges #4238); S4a-3 #4239; S4a-4 #4241; S4a-9 (the 100,000-Study benchmark) to follow S0, S1a, S1b, S3b, S3c, S3d S4a-1 to S4a-3 merged (15 October); S4a-4 open (#4241); S4a-9 benchmark pending
10 S4b Search restore — S4a
11 S5 Search UI — S4a (S4b for restore)
12 S6 Project delete and restore (flag on) — S2, S3b, S3c, S3d
13 S7 Project UI — S6
14 S8 Administrator restore — S4b, S6
15 S9 Staging configuration (cluster-gitops) — S4a; prerequisites, in order: (1) project-management deletionLifecycle on, and the web's: today only the staging API has it, so the runner's schedule never installs on staging; (2) then migrations 0005–0008 (never while PM's flag is off and the API's on); (3) #4242. Until (2), staging search deletes keep the 503. No broker grant needed (staging workloads hold .*; the narrow grants ride in broker-permissions.json for broker principals Phase ¾). Detail: rollout §4
16 S10 Soak additions (on #4010's branch) — S4a, S6, S8
17 S11 Production enablement — MVP accepted on staging; Chris's approval Never without approval

Critical path: S3a → max(S3b, S3c, S3d) → S4a → S4b → S5. S1a and S1b must merge before S4a, and S0 must be merged or agreed by then. #4062 has merged, so nothing outside this programme still gates S3c or S3d.

2. What runs in parallel

  • Now: S1b (open, #4189), S3a, S0, S2. None depends on another. S1a and #4172 have merged.
  • After S3a: S3b, S3c and S3d in parallel. They touch Bulk PDF, the import saga and the ADR-020 executor respectively.
  • Second lane: S2 → S6 → S7, alongside S4a → S4b → S5.
  • Late: S8 after S4b and S6. S9 as soon as S4a merges. S10 is written against S4a's contract and run once S4a, S6 and S8 are on its branch.
  • One worker per worktree. S1b and S4a both touch StudyRepository and the architecture tests, so they should not overlap in time with S1a still open.

3. What happens to PR #2934

Decision: supersede and close. Do not rebase it. #2934 ("durable lifecycle persistence kernel", last updated 23 September) implements the August design that Chris ruled out on 5 October:

  • It schedules physical deletion. DeletionScheduled has a DueAt and leads to Deleting, whose claim exists to erase data. It also adds permanent pmDeletedResource tombstones with anti-resurrection guards. D2 ("no actual permanent deletion") removes the need for both. The plan dropped the anti-resurrection follow-up for that reason (slices §0).
  • It uses option A (a marker plus a filter on every read), not D (detach and mark). It adds a non-null DeletionLifecycle to every Study, Project and SystematicSearch, initialised to new(). That would write a new subdocument onto every saved Study, which breaks the byte-identical rule and rewrites pmStudy as Studies are saved.
  • It is far behind main. It is based on 27 August, 5,668 commits behind, and GitHub reports it CONFLICTING. It predates the FEAT-024 fold, the ADR-020 locks and the authority programme.
  • Its local worktree holds unpushed work. pr/pr2934.bulk-pdf-deletion-persistence-kernel has two commits not on the remote: b7ae6c8418 ("preserve provisional search import cleanup") and a merge. Preserve them before the worktree is removed (D9's approach for the August work).

What to keep: the exact-token claim semantics of its DeletionLifecycle value object. That means monotonic states, renewal, takeover only after expiry, and UTC and identity validation. S3a's operation lease should reuse that logic, rewritten on pmDeletionOperation without the erasing states. Data model §4 already names this idea as worth keeping.

4. Acceptance criteria per slice

These are the criteria IDs from acceptance criteria, with the wording there (including #4172's rewording of DL-R1S-02 and DL-R1P-01). Methods: U unit, I integration on a real replica set, A architecture, E E2E on the hermetic stack, S soak drill, B Bramble benchmark, St staging check, R review.

Slice # Criterion Method
S0 1 DL-R0-09: the deletion ADR is merged with the decisions, and the August ADR-014 draft is marked superseded R
S1a 1 DL-R0-01 (Study part): with no marker, existing suites pass unchanged and an unmarked Study serialises byte-identically U, I
S1a 2 DL-R0-02: every Study write path refuses a marked Study with study-deleted (HTTP 410; hub StudyDeleted) and writes nothing. Project-scoped paths do not see a re-homed Study and also write nothing I
S1a 3 DL-R0-03 (write clause): the build fails if a Study write omits the composite predicate, or explains a miss or checks a snapshot with the lock alone A
S1b 1 DL-R0-03 (id-load clause): the build fails if a Study id load neither compares ProjectId nor is allow-listed with a reason A
S1b 2 DL-R0-03 (no-erasure clause): the build fails if a new member deletes from pmStudy, pmSystematicSearch or pmProject outside the allow-list. Until the flag is retired, the allow-list includes the flag-off legacy delete methods A
S1b 3 Each audited id-load path answers a marked Study as "not found" or study-deleted, with a test per path (data model §5.9, risk K1) U, I
S2 1 DL-R0-01 (Project part): an undeleted Project serialises byte-identically U, I
S2 2 DL-R0-04: for a deleted project, every Project*/Stage* route and hub call answers 410 to members and 404 to others, in all three authority modes. Recovery routes answer only to the owner within the window and to application administrators U, I
S2 3 DL-R0-05: a deleted project appears in no list I
S2 4 DL-R0-06: scheduled and queued work skips a deleted project. Reservation timeouts and Bulk PDF clean-up still run (S2a: the delegated-work gate; S2b: exports, daily observations, statistics jobs including scheduled maintenance, invitations, and the kept-running checks) U, I
S3a 1 DL-R0-07: monotonic states with no erasing state; exact-token leases with takeover after expiry; at most one sweeping operation per project; the window end never moves U, I
S3a 2 DL-R0-08: ledger records hold no free text U (schema)
S3b 1–7 DL-R1W-01, 02, 03, 04, 11, 14, 15 U, I, E (Bulk PDF lane)
S3b 8 Ordering (statistics owner, #4175 review): the deletion runner marks no Study until every upload of the item is cancelled (Aborting or terminal) or finished. Since S1a, the Bulk PDF delivery probe refuses a whole batch with a non-retryable StudyDeletedException if any Study in it is marked, so a delivery still in flight when the first Study moves would fail the entire batch. The test runs a delivery in flight while a search deletion starts and asserts that the runner waits, then marks I
S3c 1–5 DL-R1W-05, 06, 07, 12, 13 U, I, E
S3d 1–6 DL-R1W-08, 09, 10, 12, 13, 17 U, I
S4a 1–8 DL-R1S-01, 02, 03, 04, 08, 09, 10; DL-R1W-16 (search part) U, I, E, S
S4a 9 DL-R1S-13 (delete half): a 100,000-Study search is deleted in under 10 minutes, with save p95 within 20% of idle during the sweep (gap: no slice owned this; see §5) B
S4a 10 Re-homed write window (statistics owner, #4175 review): a fold write-window test uses the realistic re-homed shape: ProjectId set to the holding id and the marker set, not the marker alone as in S1a's test. A save whose retry reload (GetUncachedForFoldAsync, which loads by _id only) sees the moved Study is refused with StudyDeletedException, makes no second write and leaves no pending set. The test pins the _id-only reload I
S4a 11 Pending fold entries drained before marking (statistics owner, #4175 review): before the runner marks any Study, it admits the project's statistics fence (keyed by the operation id). It then settles or discards every pending fold entry on each affected Study, so no Study is re-homed with a pending set. A pending entry left on a re-homed Study would fold under the holding project or be orphaned. The sweep filter excludes Studies with PendingStatistics (data model §5.7 step 5), and Studies held back that way are retried once the fold has drained them. Verified three ways. (a) An integration test on the fold path with entries pending at deletion time asserts that those Studies wait and are moved only after the drain. (b) A query at the end of every deletion in tests and soak drills, Studies under the holding id with PendingStatistics present, returns 0 (also DL-R1S-09). © Fence diagnostics show the fence was admitted before the first mark I, S
S4b 1–3 DL-R1S-06, 07, 08 (restore half) I, E, S
S4b 4 DL-R1S-13 (restore half): the same 100,000-Study search is restored in under 10 minutes, with save p95 within 20% of idle during the reverse sweep B
S5 1–3 DL-R1S-05, 11, 12 U, E, R
S6 1–6 DL-R1P-01, 02, 04, 05, 06; DL-R1W-16 (project part) I, E, S
S7 1–2 DL-R1P-03, 07 E, R
S8 1–4 DL-R1A-01, 02, 03, 04 U, I, S, R
S9 1 Staging project-management and web carry deletionLifecycle. Staging acceptance (testing §7) passes St
S10 1 The soak's deletion drills pass, with all nine families InParity S
S11 1 Chris approves the production values change R

5. Plan gaps found while starting delivery

  1. ADR number. ADR-022 is now ADR-022-account-portal-security-hub.md, and ADR-023 is claimed by the pre-deploy migrations ADR (#4177). Resolved by #4191: the deletion ADR is ADR-024, and the README, slices S0, DL-R0-09 and D9 now say so.
  2. S1 is split into S1a and S1b. The id-load audit is larger than the plan's estimate of about 37: there are about 76 Studies.Get*/GetMany* call sites across Core, the API and the project-management consumers. The no-erasure allow-list depends on #4172's flag-off paths. S1a ships the marker and the composite guard. S1b ships the id-load and no-erasure architecture tests and the per-path fixes. Both land before S4a. S1b's count (#4189): 58 id-load call sites in 44 members (the 76 also counted project-scoped Studies.Get* reads). 27 sites already compared the project; 18 sites in 14 members were fixed, with a test per path; 13 sites in 12 members are allow-listed with reasons.
  3. Session-timeout consumers. RemoveIdleSessionConsumer, RemoveSuspendedSessionConsumer, MarkSessionIdleConsumer and CheckConnectionLivenessConsumer load the Study by id and save it. Once S4a moves Studies, a timeout for a moved Study would meet StudyDeletedException, retry and land in the error queue. The plan expects those timeouts to "fire and find nothing to release" (behaviour §9). S1b (audit) and S4a (behaviour) must make them treat StudyDeleted as nothing to do. Done in S1b (#4189) for the four consumers and the hub's disconnect, prior-study release and failed-join rollback: a marked Study (on load, reload or a refused save) is logged at Information and nothing is written. S4a's runner must close the presences of the reservations it releases, since these paths now leave them.
  4. Bulk PDF delivery to a moved Study. Since S1a, MarkBulkPdfDeliveredAsync refuses a marked Study with StudyDeletedException. BulkPdfUploadEndpoint.IsRetryableAggregateWriteFailure treats that as non-retryable, so the message goes to the error transport. The plan says the job should end Failed. S3b must map the exception to that outcome. Done in S3b-1 (#4206): the finalize consumer catches StudyDeletedException from the delivery, ends the job Failed with SearchDeletedFailureReason, writes no report and no Study, and completes without a retry.
  5. Risk-of-bias runs are not in the in-flight design. RoB is suspended, but its run store writes Studies. S1a makes a RoB rollback wait (blocked) on a marked Study rather than drop its before-image. S3b should list RoB runs as out of scope while RoB is suspended, or as waited-for. S3b-2 (#4212): out of scope while RoB is suspended. pmRobRunOperation has a lease and a phase but no ProjectId index; a RoB wait is a follow-up to add before RoB is resumed, together with the deletion it would need.
  6. No-erasure versus flag off. With the flag off, deletion erases (#4172), so D2 holds only where the flag is on. Two edges need an owner:
  7. Detached Studies are safe. A legacy erasing delete never touches them, because they carry a holding ProjectId (proved in S1a's tests).
  8. If the flag goes on → off → on, a project can be legacy-erased while one of its searches is reversibly deleted. Restoring that search must then refuse cleanly ("the project no longer exists"), and the administrator list must show it. Proposed owners: S4b and S8.
  9. DL-R1S-13 had no slice. It is split above: the delete timing and save p95 go to S4a (S4a-9), and the restore timing to S4b (S4b-4), because restore arrives only in S4b. Both run on Bramble, never on Juniper.
  10. The web needs a StudyDeleted message. S1a adds the hub reason and the 410 problem type, but the access-denied dialog and the review save effects have no copy for them yet, so they show the generic message. S5 owns the copy (behaviour §2), under the M3 contract.
  11. Staging project-management takes the chart default (false) while the staging API runs true (F4). Every consumer and job that S3a to S6 add in project-management must check its own flag value. S9 must apply the cluster-gitops change before staging acceptance, or project-management will never advance a deletion that the API accepted.
  12. The statistics owner reviewed S1a and approved its fold-path parts at e66f1ba6f, with no changes needed. Their three notes are now criteria S3b-8, S4a-10 and S4a-11 in §4. S4a, S4b and S6 still need full FEAT-024 review, as slices §3 says.
  13. The deletion indexes exist only as ADR-023 migrations (S3a). Migrations 0005 to 0008 index pmDeletionOperation and pmDeletionLedger. Nothing builds them at startup, even with Migrations:PreDeploy off, and the runner advances nothing until they exist. An environment that turns the lifecycle on must therefore run the pre-deploy migrations first: either set the chart's migrations.enabled, or run the migration entry point once. That is now part of S9 for staging, and of S11 for production.
  14. Runner versus flag (S3a). The deletionLifecycle flag's own contract (env-mapping.yaml) gates only the creation of deletions; durable work must keep converging after the flag is turned off. S3a keeps both promises. The runner's schedule is installed only where the flag is on, so production stays dark. Its consumer does not re-check the flag, so a schedule installed while the flag was on keeps finishing operations after it is turned off.
  15. Field naming (S3a). The operation stores the window as RestoreWindowMinutes plus RestoreWindowEndsAtUtc, not the data model's RestoreWindowDays. Minutes keep the local override exact. The deletion ADR should record the final names.
  16. The detached-Studies and deleted-projects gauges stay with S8 (DL-R1A-04). S3a adds only the runner's tick and per-operation counters, under the meter SyRF.ProjectManagement.DeletionLifecycle.
  17. S2 is split into S2a and S2b to keep the authority review separate from the statistics review. S2a (#4194) is the Project.Deletion state, the access gate (HTTP and SignalR, all three authority modes), the recovery-route policy, the list filters and the delegated-work denial (the evaluator denies a delegated actor's job for a deleted project, so the system-obligation path is denied in the same PR). S2b is the remaining background-job rules of restore window §7, including the statistics scheduled-maintenance skip that needs FEAT-024 review.
  18. There is no separate administrator project list. The API's project lists are the projects page (GET api/projects: My projects, public projects, pending invitations and join requests in one query), the invitation link, the recent projects of the start-up payload, and the live project-summaries feed (SubscribeToProjectSummaries). The feed decides listing in three places: the v1 hub subscription with authority off or shadow, the v1 enforced delivery, and RealtimeDispatcher. All three now share RealtimeDispatcher.ListedFor, which lists a deleted project for nobody, so every subscriber receives the payload-free delete (review on #4194). The paged FilterSort query (every access filter, including the unrelated ones an administrator view would use) has no endpoint today; S2a filters it anyway, so a future administrator list inherits the rule.
  19. ProjectRepository.GetForInvestigator is dead and cannot run. It filters on Memberships, which the class map does not map, so the driver refuses the query. Nothing calls it, so S2a leaves it unchanged; it should be deleted in a clean-up.
  20. The absent-project 404 fails after it is written. The authorization handler writes the 404 body itself, and the framework's default result handler then tries to forbid the started response ("the status code cannot be set"). This predates S2 and is reproducible through a test host. A deleted project answers non-members through the same path, deliberately, so the two stay indistinguishable; fixing it means fixing both together (follow-up).
  21. Exports and invitations have no background job to skip. A data export streams inside its own ProjectExportDataPolicy request, and its progress subscription is a ProjectExportData hub requirement, so the S2a access gate already refuses both for a deleted project. Invitation emails are sent inside the ProjectInvitePolicy request, and accepting goes through ProjectRequestToJoinPolicy. The one invitation write outside a project route, claiming invitations by email when an account registers, still runs for a deleted project, so the invitation is linked correctly if the project is restored.
  22. The statistics jobs that S2b pins. Scheduled maintenance did not check IsProjectDeleted, and S2b makes it skip (#4195). Stale repair, the backfill identity move, mode-epoch batches and daily observations already skipped, and S2b adds tests for repair and the identity move; mode epoch and daily observations were already tested. The explicit administrator maintenance and backfill routes are unchanged. Whether they should refuse a deleted project is a question for the FEAT-024 owner.
  23. S3c: the parse-ended rule is a time bound. A parse job has no durable heartbeat, so S3c-1 treats a cancelled import's parse as ended once its 5-minute job timeout plus a 2-minute grace has passed since the cancel (at once for an import cancelled while Uploading). A search deletion that cancels a running import therefore waits about 7 minutes before its clean-up check passes. A durable parse acknowledgement could shorten this; it is a follow-up, because two parse jobs of one import (a redelivered start) make a single acknowledgement unsafe.
  24. S3c: "refused at job start" means at import-job creation. New imports are refused when the saga creates the import job, not when a parse starts: an import created before the request is existing work, cancelled when it belongs to the item and waited for otherwise (recommendation R-A), and refusing its parse would cancel a colleague's import. The parse of a cancelled import stops at its own start check (S3c-1).