Rollout: flags, migration, environments and rollback

Part of the temporary reversible deletion plan. Production promotion and any production flag change need Chris's explicit approval; nothing here is that approval.

1. Flags and configuration

Setting	Kind	Default	Meaning
deletionLifecycle (existing)	Deployment feature flag (not runtime-overridable), API and project-management; add the web consumer	false	On: the DELETE routes create reversible deletions (both search routes since S4a; project DELETE keeps the typed 503 until S6). Off: production's legacy immediate deletion, which erases (owner decision 2026-10-14, see §2). Either way, Recently deleted, restore and the administrator routes keep working, and the runner keeps converging requested operations, so turning the flag off never strands data
DeletionLifecycle:RestoreWindowDays	Configuration	30 (D1)	Validated 1 to 365. Snapshotted into each deletion
DeletionLifecycle:RestoreWindowMinutesOverride	Configuration, local and E2E only	unset	Refused when the database is the production database (syrftest), the same guard the fold uses
DeletionLifecycle:ScheduleCron, BatchSize, LeaseSeconds, MaxOperationsPerTick, WaitAlertHours, PendingStatisticsWaitMinutes	Configuration	every 5 minutes; 500; 120; 5; 24; 15	Tuning. PendingStatisticsWaitMinutes (1 to 1440, S4a) is how long a sweep retries Studies held back by pending fold entries or a lock before it sets BlockedReason

There is no purge switch: nothing is ever erased (D2).

Flag decision. Flagged. Deletion changes observable semantics on a production data path, spans the API, project-management and the web, and needs per-environment rollout and a kill switch for new deletions. The flag is deployment-managed, not runtime-overridable, so all replicas agree (the same reasoning as ADR-020's flag). The existing startup rule stays: bulkPdfUpload requires deletionLifecycle, so legacy immediate deletion never runs alongside Bulk PDF.

Flag decision (2026-10-14): existing flag deletionLifecycle; OFF restores production's legacy behaviour (owner decision 2026-10-14) for promotability; the reversible lifecycle stays behind ON.

2. Migrating from the fail-closed state (#2927)

Where main is now (owner decision 2026-10-14, syrf#4172). #2927 made the three destructive routes answer the typed 503 for both flag values, which would have removed deletion from production at the next promotion. Chris decided: "keep current behaviour and place new functionality behind flag, for immediate promotability". So on main:

• Flag off (the default, and production's value): project DELETE, search DELETE and import-job DELETE perform production's legacy immediate deletion (baseline 104eca9b05), with its 200 responses and bodies, and erase what it erased with one deliberate exception: a search another project still links is only unlinked from the deleted project, never deleted, and searches linked only on the search side are handled too (the baseline deleted every search in the project's list outright). main's newer invariants apply on top: the ADR-020 typed 409 bulk-study-update-in-progress while a bulk update holds the project or search; the Bulk PDF obligation refusal on project deletion; the FEAT-024 deletion fence (ProjectScreeningBulkOperation, DeletionNamespace) around Study removal, inert where statistics are off and answering the typed 409 FamilyFenced if another statistics operation holds a family; and Studies deleted before the Project (project and import-job routes), or resumed idempotently on a retry (search route), so a failure cannot silently orphan them; a failed Study delete is logged at Error with both ids.
• Flag on: the typed 503 deletion_lifecycle_unavailable until the slices below replace it with the reversible lifecycle. Since S4a the two search routes record a reversible deletion (the contract below); project DELETE keeps the 503 until S6.

The slices then change only the flag-on branch:

• Routes. The flag-on branch of DELETE /api/projects/{id}/searches/{searchId} and …/searches/importJobs/{searchId} changes from the 503 to creating a deletion (slice S4a); project DELETE changes the same way (S6). The flag-off branch keeps production's legacy immediate deletion until production runs the lifecycle (§5).
• Contract. The routes answer 202 Accepted with a DeletionOperationDto (operation id, state, pending reason, restore-window end) instead of today's 200 with the project DTO. Regenerate the OpenAPI document and the TypeScript client (build the API in Release first); adapt the web in the same slice.

As built for the two search routes (S4a, syrf#4239). With deletionLifecycle on, after the RemoveSearch policy, in this order:

1. Fail closed. The typed 503 deletion_lifecycle_unavailable when the API has no deletion services registered, or when the deletion indexes (migrations 0005 to 0008, checked through IDeletionSchemaReadiness) are missing: a deletion recorded then would never advance. An environment that turns the flag on must run the project-management pre-deploy migrations first. The local E2E stack runs them before its services start (e2e/scripts/pm-migrations.sh, S4a's PR 4), so there the flag-on routes answer 202.
2. Idempotent repeat. An unfinished (Requested or Deleting) deletion of the same search in the project is the answer: 202 with that operation, nothing new recorded (IDeletionOperationStore.FindUnfinishedSearchDeletionAsync, served by migration 0008 ProjectId_1_State_1). This comes before the next check, so a repeat still answers after the runner has unlinked the search. Two simultaneous first requests can both record an operation (the check and the insert are not atomic); that is accepted: sweeps are serialised per project, the second finds the search already unlinked and completes with nothing moved, and later repeats answer the older one.
3. 404 (Problem Details) when the search is not in the project: linked on neither side (the project's list or the search's project ids), and, for the import-job route only, no import-job entry either. The same lookup semantics as the flag-off routes.
4. Record. DeletionOperation.RequestSearchDeletion (a new id, the requester from the current user, the clock, the reason), inserted as Requested without a lease (the runner takes one), then the ledger's SearchDeletionRequested with the requester as actor. Both routes record the same search-scope deletion; there is no separate import-job operation.
5. Run now. The API publishes IRunDeletionLifecycleCommand (best effort, 10-second bound; a failure is logged at Warning). Project-management's recurring schedule, every five minutes, is the backstop, but only where project-management's own deletionLifecycle flag is on: that flag is what installs the schedule (DeletionLifecycleScheduleService, DeletionLifecycleSchedule.cs). The API's flag does not. Where project-management's flag is off there is no schedule and nothing advances the deletion, which is why the S9 prerequisites (§4) order the flags before the migrations. No broker grant is needed on staging today: staging's API, project-management and Quartz still connect as a principal with .* on the vhost (the new syrf-staging-* principals hold the interim .* too; broker principals plan §1), so the run-now publish works and a deletion starts at once. The narrow grant matters only at broker principals Phase ¾, when workloads leave .*: the API's contract (src/services/api/broker-permissions.json) carries configure, write and read on that exchange, proved by ApiBrokerPermissionTests, and is what the definitions take then. If the publish is ever refused, it is logged at Warning and the next scheduled tick (up to five minutes later) starts the deletion, so correctness never depends on it.
6. 202 Accepted with DeletionOperationDto: operationId, scope, state, projectId, searchId, requestedBy, requestedAtUtc, reason, blockedReason, restoreWindowEndsAtUtc (null until the deletion completes). Scope, state and blocked reason are strings, as stored.

The optional body is { "reason": string }: trimmed, blank means none, at most 500 characters (400 Problem Details beyond that); it is kept on the operation and never written to the ledger. A request with no body, with or without a JSON content type, is accepted (the flag-off callers send none). No bulk-update or statistics-fence refusal applies on the flag-on path: nothing is changed yet, and those become in-flight waits. The routes declare 200 ProjectStandardDto (flag off), 202 DeletionOperationDto, 400, 404, 409 (the flag-off refusals) and 503. Because 200 is the generated client's primary answer, the client throws the parsed 202 body; the web recognises it by its shape, keeps the search and shows "Deletion requested. The search will disappear once running work has stopped."; the project's SignalR refresh drops the search once the runner unlinks it. The full UI is S5.

When the runner completes the deletion (S4a's PR 4, syrf#4241). After the Deleted write, project-management publishes SyRF.ProjectManagement.Messages.Events.ISearchDeletedEvent (ProjectId, SearchId, SearchName when it can be read, DeletionOperationId, StudiesMoved, DeletedAtUtc, RestoreWindowEndsAtUtc), once and best effort: a 10-second bound, and a failure is logged at Warning, never retried and never changes the deletion (open pages still refresh through the project notification of the unlink's project save). Each API pod consumes it on its own temporary queue api-search-deletions<32 hex> and sends the hub client method SearchDeleted to its connections subscribed to the project whose project View is rechecked at delivery; a subscriber who lost access gets nothing and is dropped. The API contract (src/services/api/broker-permissions.json) holds the queue (configure, write, read, with its _error queue) and configure and read on the event exchange, proved by ApiBrokerPermissionTests; project-management's existing SyRF.ProjectManagement.Messages.Events grant covers the publish (ProjectManagementBrokerPermissionTests). Staging needs nothing today (its workloads still hold .*); the contract strings are what the definitions take at broker principals Phase ¾. - Import-job variant. When the search or any of its Studies exists, it follows the search lifecycle. When only a failed job entry exists, with no search document and no Studies, there is nothing to keep: the entry is hidden through the same lifecycle (recorded on an operation with no Studies), not erased. - Legacy code removed — later. RemoveAndDeleteSystematicSearchFromProjectAsync, RemoveAndDeleteSearchImportJobWithStudiesFromProject, DeleteProjectAsync and StudyRepository.DeleteStudiesWithSearchAsync (whose "known window" comment is about this scheduler) are now the flag-off branch, so they stay until production has run with deletionLifecycle on and the flag is retired; only then do they go, with their tests, so no erasing deletion path remains. That removal closes #3082 and supersedes #1376. - Data. Nothing to migrate: no deletion data exists anywhere (production and flag-off environments delete immediately; staging, with the flag on, refuses). The new fields are absent on existing documents, which means "active". No new index on pmStudy is needed; the two new collections are small and get their indexes only from the ADR-023 pre-deploy migrations 0005 to 0008 (never at start-up). Slice S4a adds no migration.

3. Release order

The ADR-020 pattern: install the guards dark everywhere first, then the creators behind the flag.

Release	Slices	Effect	Gate to the next
R0: guards, dark	S0 (the ADR), S1, S2, S3a, S3b, S3c, S3d	No behaviour change: no marker, no deleted project, no operation exists; the cancel paths have no caller yet	Merged and on staging; #4062 merged before S3c and S3d
R1: reversible deletion (the MVP's user-visible part)	S4a, S4b, S5, S6, S7, S8	Delete and restore work where the flag is on; administrators can restore after the window	Staging acceptance (§4), acceptance criteria R1, the soak's deletion drills

4. Environments

1. Local and E2E. Flag on, a restore window of minutes, one-minute runner schedule. As built (S4a's PR 4): e2e/scripts/service-env.sh turns the flag on for the API and project-management and sets the one-minute schedule, and e2e/scripts/pm-migrations.sh applies the migrations before the services start (run-local.sh and the CI workflow). The restore window stays at its 30-day default until a spec needs a window end (S4b, S5).
2. PR previews. Enable through #preview-config for the PRs that need it.

3. Staging. The API already has deletionLifecycle: true; project-management and the web do not (cluster-gitops main, checked 2026-10-15), and staging has never run the deletion migrations 0005 to 0008 (no migrations key in its values). While those migrations are absent, the readiness check keeps the staging API answering the typed 503 deletion_lifecycle_unavailable for search deletes, even though its API flag is on, so merging S4a changes nothing on staging. Turning reversible search deletion on for staging is S9, with these prerequisites, in this order:

4. Project-management deletionLifecycle on (and the web's), which installs the runner's five-minute schedule (cluster-gitops change, auto-merged for staging).

5. Then migrations 0005 to 0008 applied: the chart's pre-deploy migration Job (migrations.enabled), or the project-management migration entry point (SYRF_MIGRATION_MODE) run once. Never apply the migrations while project-management's flag is off and the API's is on: the readiness check would then pass, search deletes would be accepted with no runner to advance them, and the project would refuse new imports, uploads and bulk study updates for as long as the deletion stays pending.
6. #4242: the deletion operation and ledger record the real actor, with the impersonated subject alongside.

No broker grant is a prerequisite: staging's workloads still hold .* on the vhost, so the run-now publish and the SearchDeleted fan-out work today. Their narrow grants (the API's IRunDeletionLifecycleCommand publish, its api-search-deletions queues and the ISearchDeletedEvent exchange) ride in src/services/api/broker-permissions.json for broker principals Phase ¾.

Once S9 is done, staging users can delete searches reversibly; until S5 merges, restore is available through the API only. That is acceptable on staging (seeded data, a handful of testers) and avoids extra gating. Restore window: 30 days, or 2 days during the acceptance window so testers see a window end (Q6). 4. Production. See §5.

5. The production baseline branch

• Production runs API 9.44.1 and project-management 11.43.1 from the bulk-update production baseline (104eca9b05), where both search and project deletion are immediate and erase data.
• main had the #2927 503s for both flag values, so a production release built from main before R1 would have removed deletion for production users. Since syrf#4172 (owner decision 2026-10-14) main with deletionLifecycle off keeps production's immediate deletion, so main is promotable without waiting for R1.
• Moving production from the baseline to main is a separate, approval-gated event.

Decision D7, superseded in part on 2026-10-14. The original recommendation was to cut production over to main only after R0 and R1 are accepted on staging, with deletionLifecycle on in the same promotion. The owner decision decouples the two: production can move to main at any time with deletionLifecycle off and keeps erasing deletion; turning the flag on in production (after R0 and R1 are accepted on staging) is the separate, approval-gated step that makes deletion reversible. Users keep deletion throughout (erasing before, reversible after). The rejected alternatives were (b) a cut-over with a window in which deletion answers "temporarily unavailable", and © changing the baseline branch.

When production turns the flag on no data migration is needed (§2). Production stops erasing on deletion from that moment.

Recommendation R-D: enable bulkStudyUpdateAtomicApply (version 2 bulk updates) in production before or with the cut-over, under ADR-020's own approval, so every bulk update a deletion meets can be cancelled and rolled back. With version 1 a deletion has to wait while an update is applying (in-flight work §4.3).

6. Rollback

Situation	Lever	Notes
Deletion or restore misbehaves	Set deletionLifecycle off (deployment change)	New deletions take production's legacy immediate, erasing path (owner decision 2026-10-14), not a 503: weigh that before using this lever where users can delete. Recently deleted, restore and the admin routes still work; requested operations finish (durable work converges). On staging, bulkPdfUpload must go off with it (startup rule)
A deletion is stuck waiting or sweeping	Runbook: inspect BlockedReason; resolve the job it waits for; lead the sweep forward (takeover) or run the reverse sweep	Both directions select by operation id, so partial progress is safe
A deletion waits on a search import (BlockedReason: "CancellingSearchImport" or "WaitingForOtherSearchImport")	InFlight on the operation names the import (JobId is the search id) and what holds it up (ObservedState: parse-may-run, studies-left, fence-held). A cancelled import waits about 7 minutes for its parse to end, then the runner deletes its unrevealed Studies and releases its fence itself. fence-held past that means the import's fence was not released (for example the statistics flags went off while it was held): release it with the existing fence tooling. A colleague's import is waited for until it completes or fails	The cancelled import's job stays Error, marked cancelled; nothing of it is revealed
A deletion is quarantined as CorruptRecord	Recognise it by BlockedReason: "CorruptRecord" on the pmDeletionOperation document, the runner metric syrf.deletion.runner.operations with outcome Quarantined and scope and state unknown, and an error log "is stored in a shape this build cannot read". An engineer corrects the document by hand (restore the missing or malformed field, or the state the state machine allows), then unsets BlockedReason and BlockedSinceUtc; the next tick offers it again	Only that operation stops; the runner keeps advancing every other one. Nothing is deleted. If the same record is quarantined again, compare it with DeletionOperation.FromRecord's rules before editing again
A search deletion waits on a version 2 Bulk Study Update still Planning whose attempt has died	Recognise it by an InFlight item BulkStudyUpdateWaiting with ObservedState: "Planning" on an operation whose lease has expired and is not renewed (the WaitAlertHours alert fires after 24 hours). A search deletion cannot tell whether the update is the item's work until a plan exists, and it does not plan the update itself. Re-run the operation (republish IRunBulkStudyUpdateOperationCommand for its id): the executor takes over the expired lease and plans, and the next tick decides from the plan, cancelling the update if it touches the search and waiting otherwise	A project deletion needs none of this: it cancels a Planning operation and drives the cancel itself when no attempt is live
A deletion is blocked as BulkStudyUpdateQuarantined	The deletion waits on a version 2 Bulk Study Update whose rollback found a Study changed despite its lock (ADR-020 RollbackQuarantined, with the critical "rollback quarantined" alert naming the operation). An engineer resolves that quarantine by ADR-020's procedure; the next tick re-reviews and the deletion continues	The deletion never drives or releases a quarantined update itself; its locks stay held until the engineer acts
Roll images back within main to before R0	Not while any operation is unfinished or any item is deleted	Run restore-all (admin operation) first
Roll production back to the baseline after the cut-over	Run restore-all first	Older binaries keep unknown fields in Entity.ExtraElements, so nothing fails to load; but they have no gate, so deleted projects would reappear to members, and their immediate deletion would erase data again. Re-homed Studies of deleted searches stay invisible (and unrestorable) until roll-forward
Statistics look wrong after a deletion	The usual FEAT-024 levers: fold reset, backfill, rebuild; the serving kill switch	Stale rows are never served, so the risk is performance, not wrong numbers

7. Monitoring after enablement

• Operations requested, completed, waiting (by reason) and blocked; sweep durations; waits longer than WaitAlertHours.
• Detached Studies and deleted projects over time, against Atlas storage (restore window §5).
• The staging preflight's six-hourly per-family parity for the pilot project.
• 5xx and study-deleted refusal rates on review saves (the second should be rare and short-lived).