In-flight work: cancel the job, then delete

Part of the temporary reversible deletion plan.

Decided (Chris, 5 October 2026). A Bulk PDF upload in progress is cancelled, then the deletion proceeds (D6). Running search imports and bulk study updates are cancelled too, then the deletion proceeds (D12). None of the three had a cancel path for this purpose; §3 and §4 design the two new ones.

Recommendations (marked where they appear): what to do with work that cannot be cancelled safely, and with work in the same project that does not belong to the item being deleted.

1. The rule

A deletion never fails because something is still running. The person asks once; the deletion is recorded as Requested and the deletion runner takes it from there:

1. Stop new work. From the request on, new imports, PDF uploads and bulk study updates in the project are refused at their routes and again when their job starts ("This project has a deletion in progress").
2. Cancel work that belongs to the item being deleted, at every point where cancelling is safe: Bulk PDF uploads (§2), search imports (§3), bulk study updates (§4).
3. Wait for work that has passed its point of no return, and (recommendation) for work in the same project that does not belong to the item (§5).
4. Detach only when nothing that could write to the item's Studies, or hold a statistics fence over its families, is still running. Only then is the deletion's own statistics fence admitted (§6).

While a deletion is Requested the item stays visible, labelled "Deletion pending", with the reason ("Cancelling the import of 'Search 3'", "Waiting for a bulk study update to finish").

What belongs to the item.

Deleting	Belongs to it (cancelled)	Does not belong to it (recommendation: waited for)
A search	Its own import; its PDF uploads; a version 2 bulk study update whose plan includes any of its Studies	Another search's import; a bulk study update whose plan touches none of its Studies; a version 1 bulk update (its scope is unknown until it applies, §4.3)
A project	Every import, PDF upload and bulk study update in the project	—

2. Bulk PDF uploads (D6, decided)

2.1 What the cancel path does today

POST api/projects/{projectId}/searches/{searchId}/bulkPdfUpload/uploads/{uploadId}/cancel (BulkPdfUploadController.CancelUpload, policy ProjectBulkPdfUploadPolicy) calls BulkPdfUploadJob.BeginAbort(Cancelled) and saves the Project, which embeds the job:

Job status	Cancel result today
Queued, Uploading	Accepted: status Aborting; the API tries one multipart abort at once; the upload sweeper finishes the abort and its absence proofs if that attempt fails
Uploaded, not yet claimed by the agent	Accepted: status Aborting; only the monotonic abort request is persisted, because the completed object version may already be held by the notifier. The durable notifier reconciler performs the exact-version clean-up (ADR-012 capture tombstone and hold-retirement outbox; ADR-017 receipts)
Uploaded claimed by the agent	Refused: ClaimedForProcessing ("cancellation is no longer available")
Completing, Scanning, Copying, Finalizing	Refused: InvalidState (cancellation_too_late)
Aborting (same disposition)	AlreadyAborting, answered 202: idempotent
Complete, Failed, Infected, Abandoned, Cancelled, Expired	AlreadyTerminal; an already Cancelled job answers 200

A job only reaches Cancelled once storage clean-up is proved. From Aborting onwards it can never deliver a PDF to a Study, because the abort clears the completion claim.

2.2 What the deletion does with each upload of the item

Job status when the deletion is requested	What the deletion does
Queued, Uploading, unclaimed Uploaded	Cancels it, through the same domain transition and environment check as the cancel route (extracted into a shared BulkPdfUploadCancellation service, so the route and the runner cannot drift apart)
Aborting	Nothing; it can no longer deliver
Claimed Uploaded, Completing, Scanning, Copying, Finalizing	Waits for it to end (Complete, Failed or Infected): past the agent's claim, cancelling is refused by design. A stuck Completing job is recovered by the upload sweeper after its recovery grace; I found no hard time limit on agent processing itself, so the wait carries an alert (§7)
Any terminal status	Nothing

2.3 Ordering, files and retries

• Cancellable jobs are moved to Aborting first; jobs past the cancel point are watched until terminal. A cancel that loses the race to the agent's claim (ClaimedForProcessing) joins the wait set.
• The detach starts only when every upload of the item is Aborting or terminal, so no upload can write to a Study after it moves. As a second line of defence, PDF delivery (BulkPdfUploadFinalizeConsumer → MarkBulkPdfDeliveredAsync) filters on ProjectId, so it cannot mark a moved Study; a delivery racing the move ends the job Failed without writing.
• Aborting jobs finish their storage clean-up afterwards through the upload sweeper and the notifier reconciler. They write no Study, so the deletion does not wait for them. These background paths must keep running for deleted searches and projects (they load the job from the Project's embedded list).
• Files: multipart parts of a cancelled upload are aborted by the existing abort path; a completed but unprocessed ZIP is removed by the notifier's exact-version clean-up, as for any cancelled upload. PDFs delivered by earlier, completed uploads stay linked to their Studies, move with them and come back on restore. CSV reports are kept.
• Each cancellation carries a command id derived from the deletion operation id and the upload id, so a repeat hits BeginAbort's idempotent branch. A failed first multipart abort is not retried by the deletion: the upload sweeper owns that.

2.4 As built (slice S3b-1, #4206)

• Shared cancellation. BulkPdfUploadCancellation (project-management Core) holds the environment-root check, BeginAbort(Cancelled), the optimistic save and the one best-effort multipart abort (an uploaded object is left to the notifier reconciler). BulkPdfUploadController.CancelUpload now only maps its outcome, with the same responses as before; the deletion participant calls it with the deletion's operation id and reason.
• On the upload. BulkPdfUploadJob.BeginCancelForDeletion calls BeginAbort with the command id DeletionCancelCommandId(operation, upload) (SHA-256, so a repeat is Idempotent) and records CancelledForDeletionOperationId and CancellationReason (SearchDeleted or ProjectDeleted). Both fields are written only when set (SetIgnoreIfNull). The production build (104eca9b05) has no Bulk PDF jobs at all, so it keeps the whole embedded list in Project.ExtraElements and writes it back unchanged.
• Participant. BulkPdfUploadDeletionParticipant (kind BulkPdfUpload) follows the §2.2 table. It cancels one upload per fresh, isolated read and absorbs up to three lost optimistic races per tick. It reports BulkPdfUploadCancelling (CancelRequested), BulkPdfProcessingNotFinished (Waiting, including a cancel that lost to the claim) or BulkPdfUploadNotCancellableHere (Waiting: this deployment has no storage environment authority, or the upload belongs to another environment root). It appends UploadCancelledForDeletion for every upload carrying its operation id on each review, so a crash between the save and the append is repaired. It answers "clear" only when every upload of the item is Aborting or terminal (board criterion S3b-8).
• New work refused. InitiateUpload, RetryUpload and CompleteUpload call IDeletionPendingQuery.ThrowIfPendingAsync before any storage call (409 deletion-pending). A completion replay that is already Completing or later is still answered. In project-management, where an upload's job starts, the processing claim answers RetryLater, with no claim written, while a deletion of that upload's own search or project is pending. The participant then cancels the unclaimed upload, and the redelivered claim meets the cancellation. A claim already held is never deferred, and uploads of other searches are not touched. A deferral that exhausts the agent's redeliveries leaves the upload unclaimed, which the participant cancels anyway. The query returns the oldest pending deletion only, so with two pending deletions in one project a claim may go through. The participant then waits for it.
• Delivery to a marked Study (STATUS gap 4): BulkPdfUploadFinalizeConsumer catches StudyDeletedException, ends the job Failed with SearchDeletedFailureReason, writes no report and no Study, and does not retry.
• Not yet: the job DTO does not expose CancellationReason (a swagger change; the web copy belongs to S5), and the waits of §5 and the clean-up tests are slice S3b-2.

3. Search imports (D12, decided: cancel)

3.1 How an import runs today

The saga (SearchImportJobStateMachine) moves Uploading → Uploaded → Parsing → Completed | Error; the Project's embedded SearchImportJob moves Uploading → Received → Parsing → Saving → Complete | Error. The parse job (ReferenceFileParseJobConsumer, 5-minute job timeout) admits the import's statistics fence before its first Study write (#3842) and saves Studies in batches of 200 with the project and search ids. The search document does not exist until completion: CompleteSearchImportJob creates it ("reveals" the search) in one transaction. Until then the Studies are an unrevealed import.

The terminal failure path, FailSearchImportJob, already has the properties a cancel needs:

• One owner. It commits Error on the versioned Project; a completion that arrives later finds Error and refuses ("Terminal failure already owns this import"), which itself runs the failure clean-up.
• Clean-up. It deletes the unrevealed Studies (DeleteSystematicSearchStudiesAsync) and releases the import's fence as Stale. A replay resumes the clean-up idempotently.

Keeping that clean-up is decided (Q7): those Studies were never shown, so removing them is clean-up of a failed upload, not deletion of user data, and does not conflict with "no actual permanent deletion".

3.2 The new cancel path

CancelSearchImportJobAsync(projectId, searchId, deletionOperationId) in project-management:

1. Claim the outcome. In one transaction, read the import job and the search document. If the job is Uploading, Received, Parsing or Saving and the search is not revealed, mark it Error with a typed reason CancelledForDeletion and the deletion operation id (new additive fields on SearchImportJob), and commit. This uses the same versioned write as the failure path, so cancel, failure and completion have exactly one winner.
2. Stop the parse promptly, or never start it. The parse checks the job status when it starts and again before each batch (one keyed Project read per 200 Studies), whatever the statistics flags, and stops when it is Error. It then reports a fault, as for any cancelled parse. (#4062's parse-start guard does the same check, but only when a statistics fence is requested.) Without these checks the parse would stop at its 5-minute job timeout and the failure clean-up would still remove what it wrote; the checks make it seconds and keep a cancelled import from writing at all.
3. Clean up through the existing path. The fault reaches the saga in Parsing, which runs FailSearchJobActivity → FailSearchImportJob. That finds the job settled (Error), deletes any unrevealed Studies left (including a batch saved just before the parse saw the cancel) and releases the import's fence. The deletion runner also calls the same replay itself once the parse has stopped, so the clean-up never depends on a message arriving.
4. Then delete. The runner waits until the parse job has ended, no unrevealed Study of the search is left and the import's fence is released. The search is then deleted as the "import-job variant": the failed job entry (marked cancelled) moves to Recently deleted and can be restored as an entry, with no Studies, like any failed import.

Import status when the deletion is requested	What the deletion does
Uploading (browser still sending the file)	Cancel: Error (CancelledForDeletion). If the file lands later, the saga starts a parse that stops at its start check (the job is already Error); nothing is written
Received (file in S3, parse not started)	Cancel: same; the parse stops at its start check
Parsing, Saving (Studies being written, unrevealed)	Cancel: Error, parse stops at its next batch, failure clean-up removes the unrevealed Studies and releases the import's fence
Completion in flight	One winner: if completion commits first, the search is revealed and the deletion detaches it like any revealed search; if the cancel commits first, completion refuses and cleans up
Complete (revealed)	Nothing to cancel; the search is deleted normally
Error	Nothing to cancel

What is kept. The uploaded reference file in S3 is kept (as for any failed import today). The saga state is left as it ends (Error), unchanged in shape. The import-job entry is kept, marked cancelled.

As built (S3c-1, #4205).

• IProjectManagementService.CancelSearchImportJobAsync(projectId, searchId, deletionOperationId, cancelledAtUtc) reads the Project and the search in one search transaction (the same uncached reads and versioned SaveWithoutDispatchAsync Project write as completion and FailSearchImportJob). It answers Cancelled, AlreadyCancelled (any repeat, by any operation: the first cancel owns the import), Revealed (completion won), Settled (completed or failed on its own), NotFound, or Conflict (a competing write won the transaction; nothing written, the next tick decides again). Only Cancelled writes.
• The job gains four optional fields, absent unless a deletion cancelled it: CancellationReason (CancelledForDeletion), CancelledByDeletionOperationId, CancelledAtUtc and CancelledFromStatus. Its error text is "Import cancelled: the search or project is being deleted." Build 104eca9 reads them into ExtraElements and writes them back unchanged (SearchImportJobCancellationCompatibilityTests).
• The parse reads the job uncached at its start and before every batch of 200 (GetUncachedOrDefaultAsync, one keyed Project read per batch), whatever the statistics flags, and stops on any settled job (Complete or Error) with a faulted result. A parse that stops records nothing on the job, and progress and parse results are never written to a job a deletion cancelled. With the flags off this also stops a redelivered parse start of a completed import, which used to parse again.
• When the parse has ended. A parse job has no durable heartbeat, so the end of the one batch whose check passed before the cancel cannot be observed directly. SearchImportParseLimits.HasParseEnded treats the parse as ended at once for a job cancelled while Uploading (no parse can have passed its start check), and otherwise once the parse job's timeout (5 minutes, shared with ReferenceFileJobConsumerDefinition) plus a 2-minute stop grace has passed since the cancel. A search deletion that cancels a running import therefore waits about 7 minutes.
• GetSearchImportCleanupStateAsync reports the three conditions of step 4 (parse ended, no unrevealed Study of the search under the project, no active fence for the import's operation id, read whatever the flags). SettleCancelledSearchImportAsync runs the FailSearchImportJob replay itself once the parse has ended and the clean-up is not finished, then reports again; it never runs while the parse may still write.

As built (S3c-2, #4214).

• SearchImportDeletionParticipant (kind SearchImport, registered by SearchImportDeletionRegistry in project-management only) reads the Project uncached on every tick. A search deletion owns that search's import; a project deletion owns every import. An import of another search that is still in flight is reported Waiting (WaitingForOtherSearchImport) and never cancelled (R-A). An owned import that is in flight is cancelled; one that completed or failed on its own needs nothing; a cancelled one is reported CancelRequested (CancellingSearchImport, ObservedState naming what is left) until SettleCancelledSearchImportAsync reports it cleaned up.
• Each import this operation cancelled gets one ImportCancelledForDeletion ledger record: the deleting person as actor, the search id as subject, a deterministic id, so a repeated tick writes nothing.
• New imports are refused while a deletion or restore is pending in the project (IDeletionPendingQuery.ThrowIfPendingAsync, 409 deletion-pending): at the signature route (POST api/projects/{projectId}/searches/getSignature), before anything is signed, admitted or published, and when the saga creates the import job (CreateSearchImportJobActivity), so the saga ends Error with no job written. Neither check is flag-gated. A parse start is not refused: an import created before the request is existing work (STATUS §5 gap 16).

4. Bulk study updates (D12, decided: cancel where it is safe)

4.1 Version 2, all or nothing (ADR-020)

Phases: Planning → Planned → Locking → Applying → Committed → Releasing → Terminal, or on a failure, denial, cancellation or the ceiling RollingBack → RolledBack → Releasing → Terminal; RollbackQuarantined needs an engineer. The executor (BulkStudyUpdateExecutor) already honours a cancelled token cooperatively: before each apply batch and once more before commit it rolls back with outcome Cancelled; an attempt that takes over an Applying operation always rolls back (ADR-020 D2). What it lacks is a way for another process to ask.

New: a durable cancel request. BulkStudyUpdateOperation.CancelRequest (RequestedAtUtc, DeletionOperationId), set by a conditional write. The executor treats it exactly like a cancelled token: StopReasonAsync, the lock phase and the pre-lock phase check it, and so does every attempt that takes over. Then:

1. The deletion runner sets the request.
2. If an attempt holds a live lease, it sees the request at its next batch (seconds) and rolls back.
3. If no attempt is live (the job is queued, or its attempt died), the runner acquires the operation's lease itself, as an attempt owned by the deletion (deletion:{operationId}), and runs the executor with a cancelled token. Every phase then does what ADR-020 already specifies, under its own lease, generation guard and crash recovery.

Phase when the deletion is requested	What cancelling does	Kept or undone
Planning, Planned	Ends Terminal with outcome Cancelled; no lock was taken	Nothing was written
Locking	Releases every lock taken (the existing refusal path), ends Cancelled	Nothing was written
Applying	Stops before the next batch, rolls back every applied batch from its before-images, releases, ends Cancelled	Every change undone
Committed	Too late: the point of no return has passed. The deletion waits for the release	The update is kept, and moves with the Studies
RollingBack, RolledBack, Releasing	Already undoing or finishing; the deletion waits	As the operation decides
RollbackQuarantined	Cannot finish: a Study changed despite its lock. The deletion is blocked with BulkStudyUpdateQuarantined until an engineer resolves it (ADR-020's existing procedure)	—
Terminal	Nothing	—

The rollback runs under ADR-020's approved P9 system obligation bulk-study-update-rollback.v1 (restore and release only), so it needs no one's personal authority.

As built (S3d-1, #4207).

• MongoBulkStudyUpdateOperationStore.RequestCancelAsync records the request with one conditional write.
• It never changes the generation or the lease, so a live attempt is not fenced.
• It never writes a Terminal record.
• A repeat keeps the first request.
• The field is absent until a deletion asks, and the store maps it explicitly.
• The executor checks the request:
• at the start of the pre-lock phases (Planning, Planned);
• at the start of locking, before every lock batch and before the statistics-fence admission;
• before every apply batch (StopReasonAsync);
• in every takeover of Applying, which rolls back with outcome Cancelled, not Failed.
• The commit write itself refuses an operation that has a request, so a request recorded during the last batch still rolls it back.
• A cancel while Locking releases the statistics fence before it releases the locks. An earlier attempt may have admitted the fence and died before Applying. Releasing a fence that was never admitted writes nothing.
• BulkStudyUpdateCancellation.CancelForDeletionAsync is steps 1 to 3 above.
• It records the request.
• If any attempt holds a live lease, it returns. This includes an earlier run of the same deletion.
• Otherwise it runs the executor as deletion:{deletionOperationId} with a cancelled token. It never acquires a RollbackQuarantined or Terminal operation.
• The job's owner sees "The bulk update was cancelled because its project or search is being deleted. Nothing was changed."
• Tests run on a replica set (BulkStudyUpdateExecutorCancelTests, MongoBulkStudyUpdateOperationStoreCancelTests).
• Each row of the table above has a test.
• Every rollback is checked against its before-images.
• DL-R1W-12: a stalled attempt cannot admit its fence after the deletion's takeover.
• DL-R1W-13: a deletion that dies mid-rollback is taken over at the next generation, and the rollback completes.
• The deletion participant that calls this came with S3d-2 (§4.3).

4.2 Search deletion and version 2 scope

A version 2 update belongs to a deleted search when its plan includes any of the search's Studies (one query over the planned Study ids, at most 10,000). While it is still Planning the runner waits the few seconds until the plan exists, then decides. Updates that touch none of the search's Studies are waited for (recommendation, §5).

4.3 Version 1

Version 1 is what staging and production run today: bulkStudyUpdateAtomicApply is unset in both, so it defaults to off. Version 1 validates the file (reads only), then applies it row by row with no locks, no before-images and no rollback (ParseBulkStudyUpdateJobFile), under a 5-minute job timeout; it admits and releases its statistics fence in one call.

Version 1 status	What the deletion does
UploadingToS3, ValidatingOnServer (nothing written yet)	Cancel (project deletion): mark the job ErrorValidatingOnServer with reason CancelledForDeletion; the job checks the status before it admits its fence and starts applying, and stops
Parsing (applying)	Wait. Stopping it would leave a partial update with no way back, which is the failure ADR-020 exists to prevent. The wait is bounded by Version1StaleAfter: at most 30 minutes since the job last changed, after which the job is treated as abandoned and a BulkStudyUpdateAbandonedForDeletion ledger record is written (Chris, 14 October 2026, option A)
Terminal	Nothing

For a search deletion a version 1 update is waited for in every status (recommendation): its scope is unknown until it applies, so cancelling it could discard an update to other searches.

Recommendation R-D. Enable bulkStudyUpdateAtomicApply (version 2) before the production cut-over (D7), so every bulk update in production can be cancelled cleanly. That is its own approval under ADR-020's rollout.

As built (S3d-2, #4213). BulkStudyUpdateDeletionParticipant (kind BulkStudyUpdate) is the runner's in-flight participant for §4.1 to §4.3.

• Version 2, project deletion. It cancels every unfinished operation in the project through BulkStudyUpdateCancellation. A version 2 job that its start has not handed off yet has no operation record. The participant creates the same record the hand-off would create (BulkStudyUpdateOperation.ForJob, idempotent), so whichever comes first, the job's own start finds the cancel on it.
• Version 2, search deletion. An operation still Planning is waited for. One that is Planned, Locking or Applying is cancelled when its plan includes any of the search's Studies: the planned Study ids from the plan chunks, then one pmStudy query on _id, ProjectId and SystematicSearchId. Any other operation is waited for (R-A). An operation this deletion has already asked stays asked.
• Version 2 outcomes. A cancelled operation is CancelRequested (BulkStudyUpdateCancelling) until it is terminal. Committed, RollingBack, RolledBack and Releasing are Waiting (BulkStudyUpdateWaiting). RollbackQuarantined is Waiting with BulkStudyUpdateQuarantined, listed first so that it becomes the deletion's blocked reason. This holds in either scope (fail closed).
• Version 1 and the held legacy jobs, project deletion.
• A job in UploadingToS3 or ValidatingOnServer is cancelled with BulkStudyUpdateJob.CancelForDeletion inside the versioned Project save: it becomes ErrorValidatingOnServer, with CancellationReason CancelledForDeletion and CancelledForDeletionOperationId. From then on, the job's moves to validating, to applying and to completion refuse with BulkStudyUpdateCancelledForDeletionException. That is the status check in the move to applying, which comes before the statistics fence is admitted. A late failure report keeps the cancellation.
• A job that is applying (Parsing) is waited for.
• Version 1, search deletion. Every live job is waited for (R-E).
• Version 1 liveness. A job counts as live while its upload window is open, or while its LastModified is within Version1StaleAfter (30 minutes: the 5-minute job timeout plus four retries). An upload whose window is unknown (a legacy job persisted without a creation time) is bounded the same way, by its LastModified, so it cannot hold a deletion for ever. A job that stopped moving longer ago than that can write nothing more, because no retry can start while a deletion is pending. The deletion does not wait for it. This covers the held legacy jobs stuck in Parsing since September 2026. Chris chose this bound on 14 October 2026 (option A).
• Abandoned version 1 jobs. An unfinished version 1 job that is no longer live, and that no deletion cancelled, is treated as abandoned. The participant logs a warning (job id, status, last change) and appends one BulkStudyUpdateAbandonedForDeletion record, with the deleting person as actor and the job id as subject, on every review that sees it (deterministic id, so it is written once); the warning is logged only when the record is written. Nothing on the job is changed or erased.
• Scope of the records. A record is written only for a job the deletion would otherwise have waited for. In a project deletion that is a job applying (Parsing); its uploading and validating jobs are cancelled, not abandoned. In a search deletion it is every unfinished version 1 job in the project, because recommendation R-E waits for all of them: a version 1 job's scope is unknown until it applies, so any of them could touch the search. Project-wide records are therefore the same set the search deletion waits for, not a wider one.
• Ledger. Every cancelled update gets one BulkStudyUpdateCancelledForDeletion record, with the deleting person as actor and the job id as subject. The record is appended on every review that sees the cancel, with a deterministic id so a repeat writes nothing. A version 2 operation that is already terminal is no longer reviewed as unfinished, so every review also appends for each operation whose cancel request names the deletion and whose outcome is Cancelled (FindCancelledForDeletionAsync). A failed append is therefore repaired by the next review, even when the operation, or its own attempt's rollback, finished in between. Version 1 is repaired the same way, from the job's CancelledForDeletionOperationId.
• New work is refused while a deletion is pending, at the bulk-update signature route (StudyController.GetBulkStudyUpdateS3RequestSignature, 409 deletion-pending) and at the job start (StartBulkStudyUpdateJobConsumer, not retried). Neither check is flag-gated.
• During a project deletion, the participant cancels the job.
• During a search deletion, a job whose start was refused stays unstarted until its upload window expires. The person uploads it again after the deletion.
• Registration. DeletionInFlightBulkStudyUpdatesRegistry.

5. Work that is waited for (recommendation)

Work	Why it is not cancelled	Bound
Bulk PDF processing past the agent's claim	Refused by the upload protocol	No hard limit found; alert after 24 hours
A version 2 update in Committed, RollingBack or Releasing	Past its point of no return, or already undoing	Release takes seconds
A version 1 update while applying	No rollback exists	30 minutes since the job last changed, then abandoned: logged and in the ledger (Chris, 14 October 2026, option A)
Another search's import, during a search deletion	Not the item's work; cancelling it would discard a colleague's upload	5-minute parse timeout, then completion
A bulk update that does not touch the deleted search	Not the item's work	30-minute ceiling (version 2); version 1 at most 30 minutes since the job last changed, then abandoned and recorded in the ledger
Agreement-threshold recalculation, question deletion	Project-wide rewrites with their own owners	The recalculation job
A data export of the project	Read-only; it finishes quickly	Ignored after a 1-hour staleness bound
Another deletion or restore in the project	Serialised by design	Sweeps take minutes

Alternative for the second half of this table: cancel unrelated imports and bulk updates as well. It is simpler to explain but discards work that has nothing to do with the deletion. Chris to confirm.

5.1 As built (slice S3b-2, #4212)

Each wait reads existing durable state only. None adds a field, an index or a collection.

Kind	Participant	Waits while	Bound
Recalculation	RecalculationWaitParticipant	Project.ActiveInclusionInfoCalculationJob is set (any statistics flags); or the statistics control row holds InclusionRecalculationToken (RecalculationRunning) or DefinitionRewriteToken (QuestionDeletionRunning), which are written only with the statistics flags on	None: no lease exists. A stranded marker keeps the deletion waiting, and the blocked alert fires after WaitAlertHours (FEAT-024's stranded-fence diagnostics are the runbook)
DataExport	DataExportWaitParticipant	A pmDataExportJob of the project is InProgress with Audit.LastModified within the last hour. A Pending export has not started reading	The planned 1-hour staleness bound (Q4), implemented. A record left InProgress by a dead process stops counting after an hour
OtherDeletion	OtherDeletionWaitParticipant	Another operation of the same project is Deleting or Restoring, or was Requested earlier (ties broken by id, so the order is total and two never wait for each other). Reason ProjectSweepBusy	The other operation's own progress

Not observable (plan gap): a question deletion with the statistics flags off, and the administrator's synchronous inclusion recalculation routes (update-study-inclusion, update-all-study-inclusion). They run inside one HTTP request and write no durable "running" marker, so there is nothing to wait for. They are short, and the deletion's own sweep (S4a) and the Study write guard are the backstop. Risk-of-bias runs: out of scope while RoB is suspended (STATUS gap 5).

Clean-up of cancelled uploads (DL-R1W-04). The upload sweeper and the notifier's cleanup registration load the upload from the Project's embedded list by id, and never ask whether its search is still linked. Tests pin both after the search is unlinked. The project-deletion side (both keep running for a deleted project) is pinned by S2b (#4195). A cancelled upload holds the usual 30-day retry authority, so the exact-version clean-up of an uploaded, cancelled object registers once that window has lapsed, as for any cancelled upload.

6. Statistics fences and late admissions

Each cancelled job owns a fence over the project's families: the import's (SearchImportNamespace, keyed by search id) and the bulk update's (BulkStudyUpdateNamespace, keyed by job id). The ordering keeps the fences from ever overlapping:

1. The cancelled job releases its own fence through its own path: the import's failure clean-up, or the bulk update's rollback and release (version 1 never admits once cancelled before applying).
2. The deletion runner admits the deletion's fence only after every cancelled or awaited job is terminal and its fence is released. The fence is never held while waiting.
3. No late re-admission. Since #4062 (open; a prerequisite of slices S3c and S3d), an admission under a reused operation id re-activates a cleared fence. A cancelled job must therefore never admit again after the deletion's release. Two guards cover it:
4. Imports: #4062's parse-start admissionGuard refuses an import whose job is already Complete or Error, and the parse's own start check (§3.2) stops it before any write; a cancelled import is Error, so a redelivered parse start admits nothing and writes nothing. A late failure replay re-admits only while unrevealed Studies of the search remain under the project id; after the clean-up and the detach there are none, so it admits nothing.
5. Bulk updates: #4062 guards the version 2 admission at the attempt's own generation, and the runner's takeover advances the generation, so a stalled attempt cannot admit late. A Terminal operation is never acquired again. Version 1 checks the cancel before admitting.
6. A test pins each: after the deletion's release, a redelivered parse start, a late failure replay and a stalled bulk-update attempt leave no active fence (fence diagnostics show none). As built for imports (S3c-1): SearchImportCancelForDeletionTests checks, with the fence diagnostics, that after the clean-up a redelivered parse start and a late FailSearchImportJob replay leave no active fence. The parse's own uncached start check now stops a settled import before #4062's admission guard is reached, with the flags on or off.

7. Idempotency and crash recovery

As built (S3 seam, #4203). The runner's Requested-stage handler (DeletionInFlightStageHandler) asks each IDeletionInFlightParticipant in turn on every tick, renewing the lease between them. Each participant cancels the item's own cancellable work idempotently, appends its ledger records and returns what is still outstanding; the handler records the combined answer on the operation. Slice S3b adds the upload participant and the waits for recalculations, exports and other deletions; S3c the import participant; S3d the bulk-update participant. New work is refused at routes and job starts through IDeletionPendingQuery (409 deletion-pending, problem type deletion-pending), which reads migration 0008's index.

As built, the hand-off (S4a-2). For a search deletion, the tick whose review finds nothing outstanding does not stop there: in the same tick, so that review is the proof that every cancelled or awaited job is terminal and has released its own fence (statistics fix 6), ISearchDeletionSweepStart reads the link snapshot, admits the deletion's fence and moves the operation to Deleting in the same write as the review. The fence is never held while waiting: a busy fence leaves the operation Requested, blocked StatisticsFenceBusy, and a taken sweep slot releases the fence again. A project deletion still only records that it is cleared (slice S6 owns its next step). See data model §5.7.

• Every cancel is a conditional write that is a no-op when repeated: the import's Error claim, the bulk update's CancelRequest, the upload's BeginAbort with a derived command id.
• The deletion operation records each cancelled or awaited job and its last observed state (CancelledJobs, AwaitedJobs). After a crash, whoever takes over the deletion's lease re-reads every job and continues: claimed jobs are not claimed twice, finished ones are skipped.
• A runner that dies while driving a bulk update's rollback leaves that operation with an expiring lease; the next deletion tick, or the bulk update's own job retry, takes it over and continues the rollback (ADR-020's takeover).
• A wait longer than WaitAlertHours (proposed 24) sets BlockedReason on the deletion and raises an alert. A deletion never skips a wait.

8. Authority and audit

• Who cancels. The system, acting for the deletion. The deleting person's authority (RemoveSearch, or project owner) was checked when the deletion was requested; the cancellations are its consequence, so they do not need the person to hold the Bulk PDF, import or bulk-update permissions. Bulk-update rollback runs under the P9 obligation above; the import clean-up is the existing system compensation.
• Audit. The deletion ledger records one value-free entry per cancelled job: UploadCancelledForDeletion, ImportCancelledForDeletion, BulkStudyUpdateCancelledForDeletion, with the deletion operation id, the job id, the deleting person as initiating actor and system as executor. A version 1 bulk update the deletion stopped waiting for gets one BulkStudyUpdateAbandonedForDeletion entry of the same shape (§4). Each cancelled job also carries the deletion operation id on its own record.
• What the job's owner sees. "Import cancelled: the search was deleted." "Bulk study update cancelled: the project was deleted. No changes were applied." The upload shows Cancelled with the reason "The search was deleted".
• What the deleting person sees. The delete dialog lists what will be cancelled (proposed copy): "1 import in progress and 1 bulk study update will be cancelled. 1 PDF upload will be cancelled; PDFs already added to studies are kept."

9. Tests and acceptance

• Criteria DL-R1W-01 to DL-R1W-16 in acceptance criteria.
• Unit: each status table above; idempotent repeats; the "belongs to the item" rule; the wait set; new-work refusals at the route and at job start.
• Integration (real Mongo, with RabbitMQ for the job paths):
• an import cancelled in each status; a cancel racing completion (both orders); a parse batch saved just before the cancel is cleaned up; no unrevealed Study left; the saga ends Error;
• a version 2 update cancelled in each phase, with the before-image check that every applied change is undone; a cancel at Committed waits and keeps the update; a quarantined rollback blocks;
• version 1 cancelled while validating; waited for while applying;
• the late-admission tests of §6;
• crash and takeover at every step, including the runner dying mid-rollback.
• E2E: delete a search while its import runs; delete a project while a bulk study update runs; the Bulk PDF lane spec for uploads (testing §4).
• Soak: drills searchDeleteDuringImport and projectDeleteDuringBulkUpdate (testing §5).