Capacity-write transaction admission

Issue #3185 closed two pre-staging requirements: durable-mode disagreements must produce a typed HTTP response rather than a 500, and capacity-sensitive source writes must read the durable mode in their own transaction. Issue #4286 superseded the first for review writes: a disagreement between this replica's ActiveReviewerTrackingAvailable and the recorded durable mode no longer refuses the write at all (see Durable-mode disagreement); only an unverifiable mode still produces the typed 503. The existing allocation eligibility, atomic capacity filters, reservation pipeline and optimistic version checks remain the source-write contract.

When materialized writes are enabled, standalone assignment and screening capacity saves own a snapshot transaction. The annotation capacity save never owns one: its only caller (SubmitAnnotationSessionService) opens the transaction that also carries the presence rows and the annotation delta, and the durable-mode read joins it. Calling it without a session while the flag is on fails closed rather than reading the fleet-wide mode outside the write's snapshot. The statistics-enabled screening overload checks the durable mode in its existing source-and-statistics transaction before preparing or replacing the Study. A caller whose assumed mode no longer matches the verified one aborts before source mutation. A rejected statistics commit aborts the preceding source replacement. Each request captures MaterializedProjectStatisticsWrites once, at entry, and passes that value to both the transaction decision and the durable-mode agreement check. RuntimeFeatureFlagProvider.ApplyBackendFlags mutates the shared FeatureFlags instance in place every thirty seconds, so re-reading the property at the second decision point could admit a write whose mode read never joined a snapshot, or refuse one for a flag change that arrived after the transaction was already chosen. One request sees one value.

A capacity guard that matched nothing aborts its transaction and, for the screening save, runs its diagnosis outside it, matching the statistics overload: nothing was written, so nothing is committed. An assignment filter miss is the ordinary "already reserved, already completed or at capacity" answer, so it also leaves the repository's Study cache untouched; only a committed assignment evicts. The screening save evicts however its attempt ends, because PrepareScreeningSave runs OnSaving on the cached instance before Mongo is called. Every statistics-writing transaction — the owned capacity transactions and the three overloads that open their own (SaveScreeningWithCapacityGuardAsync with a commit, SaveWithAnnotationStatisticsAsync and TrySaveScreeningWithStatisticsAsync) — starts at ReadConcern.Snapshot from one shared TransactionOptions, so a path's durable-mode read, its capacity guard and its source replace cannot land on different committed snapshots. They all commit through the shared bounded unknown-commit-result retry, so a partition that returns UnknownTransactionCommitResult retries the idempotent commit instead of surfacing as a 500.

MongoDB's TransientTransactionError label means the transaction aborted, whichever half raised it — an unknown commit result carries UnknownTransactionCommitResult instead and has already been retried by the shared helper — so no answer given for one can hide a durable write. What decides the response is therefore whether the callback may be re-run, not which half failed. Assignment builds its filter and pipeline from immutable arguments, so it is retried in a fresh snapshot for at most three whole-transaction attempts; when that budget is spent it answers its ordinary null rather than raising a driver exception at the client, because a caller cannot tell a lost race from a full stage. A mutable Study callback has already applied OnSaving and a classified transition, so replaying it would apply them to newer source data: it is never re-run, and answers the caller's existing optimistic-concurrency result (VersionStale or false) for the caller to reload and reapply. The screening statistics paths carry a second net: TrySaveScreeningAsync mints its operation envelope once outside its retry loop, so a replay is rejected as DuplicateOperation and treated as saved.

With materialized writes disabled, these standalone paths create no statistics transaction and do not read durable controls. This preserves the default deployment's write behavior and hot-path cost. Which overload a caller picks is the one capacity decision no in-transaction check can repair: choosing the plain optimistic save from a stale cached flag commits a screening with no capacity guard, and the chosen overload cannot add the guard back. ReviewController.TrySaveScreeningAsync and SubmitAnnotationSessionService therefore branch on ResolveVerifiedActiveReviewerTrackingModeAsync, which runs the same fail-closed agreement check. That resolution is gated on materializedProjectStatisticsWrites alone, not on whether a statistics commit was prepared: the family flag and the project allowlist decide whether a delta is written, not whether this replica's capacity semantics are the fleet's, so a project outside the allowlist would otherwise still save unguarded on a stale replica.

That resolution happens once per attempt, not once per request: the untracked overloads refuse when the mode has moved under them, and re-entering the same branch on every retry would spend the whole budget on a conflict no reload can resolve. The same rule governs which mutation a deletion applies, since restoring a reviewer's slot reservation is a capacity decision too.

Both untracked statistics saves — screening and annotation — re-check inside their transaction and refuse when the mode they read there says tracking is available: between the caller's resolution and the transaction, the durable mode and the process flag can both move to "tracked", which the agreement check accepts because they now agree, and the version-only replacement would commit unguarded. Nothing has been written, so they answer the optimistic-conflict result and the caller reloads, re-resolves and reapplies. The comparison is against the mode the caller assumed, not merely against "tracked": a reconciliation submission and a session deletion need no capacity guard, but both were still shaped by that mode — a deletion chooses whether to restore the reviewer's slot reservation from it — so a moved mode makes their decision stale too. RemoveSession therefore carries its own bounded reload loop that re-resolves the mode and re-chooses the mutation.

A save with no delta to write — the family off, or the project outside the allowlist — still goes through the transactional overload while writes are on, with a null statistics commit: Prepare returns null, so CommitAsync is never invoked and the request pays the version/mode admission but advances none of the project control row's three clocks (the write-contention benchmark records this population as no_delta_saves, #3255). That includes the session deletion. assumedTrackingMode has no default, so every caller states the mode it shaped its write from rather than inheriting one. What the family flag and the allowlist decide is whether a delta is written, not whether the mode the caller branched on is still the fleet's when the replace happens, and the plain SaveAsync / TrySaveExistingAsync calls re-read nothing. With writes off those calls are unchanged and open no transaction.

Every caller-side early return that hands a reviewer a study with no reservation follows the same rule, because the atomic assignment's own in-transaction gate is never reached when the caller returns before it: the direct and random assignment paths in StageReviewService and the JoinStudyReview grant in NotificationHub all resolve the verified mode when writes are on, and read nothing when they are off. Direct navigation goes further, because it is the one grant that both decides "no reservation is needed" and reads the study it hands over: TryLoadStudyWhenUntrackedAsync takes both from one snapshot, so no window remains between them.

The random-assignment entry points also resolve the mode before their pre-loop existing-reservation lookup, and re-run that lookup inside the loop when a skipped lookup is followed by a verified tracked answer — otherwise the reviewer would be given a second slot. The direct-request controller path decides whether to run the guarded direct-navigation method from the same verified mode.

JoinStudyReview takes the same snapshot-bound treatment, because its untracked grant creates no reservation, connection or presence and so has nothing later that would revalidate the mode. The direct-request controller path runs the guarded method whenever writes are enabled rather than deciding from a standalone read of its own, and an indeterminate annotation commit evicts the Study cache before propagating, since that commit may already be durable.

What remains a recorded residual for #3308 is narrower: the three random-assignment loops hand back a candidate loaded by a preceding query, so re-reading it inside the mode's transaction would be a different document. The mode they decide from is verified and a late existing-reservation lookup stops a transition from granting a second slot, but a transition after the final decision is not observed by any later operation on those paths.

The same replay-safe callbacks (assignment and typed activity admission) also re-run, within that three-attempt budget and after a short jittered backoff, when their statistics half is rejected for a transient reason — a lost compare-and-set or a moved epoch, lease or fence (#3740). The rejection aborted the transaction, so the next attempt re-reads the source and mints a new operation envelope. A permanent reason, or the last attempt's rejection, propagates as the typed rejection it always was.

An indeterminate commit — the bounded unknown-result retry spent — may already be durable, so the capacity-write wrapper evicts its cache before rethrowing; an abort or a filter miss keeps the cache, because neither can have changed anything. Callers that own their transaction commit through the same bounded retry, and a caller that replaces SaveAsync with a repository statistics overload dispatches the aggregate's domain events itself after a confirmed commit, because the overload cannot.

Every review action that can reach one of these writes declares the typed 503, and RemoveSession declares its optimistic 409, so the generated client models both instead of treating them as unexpected errors. The API maps only ProjectStatisticsDurableModeDisagreementException to application/problem+json, HTTP 503, with code statistics_durable_mode_disagreement; unrelated failures retain their normal handling. Since #4286 a review write raises it only when its host cannot verify the durable mode at all (statistics writes on, no gate registered: a miswired host), which is a deployment fault the caller cannot fix by resubmitting, so 503 rather than 409 remains the honest answer.

Durable-mode disagreement (#4286)

Until #4286 a disagreement between this replica's ActiveReviewerTrackingAvailable and the mode recorded on the singleton control row threw the typed 503 from every capacity write. Because the flag is fleet configuration and nothing in production moves the recorded mode (the two-stage transition, matrix M15, has no owner), changing activeReviewerTrackingEnabled or signalRActive while a singleton existed made every claim, screening save and annotation save in every project, allowlisted or not, answer 503 until the flag was changed back.

A disagreement is a statistics bookkeeping state, not a fault in the reviewer's work, so the source write now proceeds and the statistics refuse instead:

• The write follows this replica's flag. ResolveVerifiedActiveReviewerTrackingModeAsync and the in-transaction check return the durable mode while the process flag agrees with it, and the process flag (captured once, before the read) on a disagreement. Those are the capacity semantics this replica's hub, allocation reads and the writes-off path already apply, so a flag flip with statistics writes on behaves exactly as it does with them off. The assumed-mode refusal on the untracked overloads is unchanged: a caller that resolved one mode and finds another reloads.
• The statistics fail closed, through code that already existed. The annotation writer compares the same two values and routes the transition source-only, staling its families; a fold entry records the mode it was written under and the fold stales on a mismatch; the screening families read no tracking input. Every authoritative and materialized read answers DurableModeDisagreement, and rebuilds, scheduled repair, drift checks and daily observations skip, until the flag and the recorded mode agree again.
• Logged once per state. The repository logs one warning per distinct (process flag, recorded mode, mode epoch) it observes, naming both values and the epoch.

Operator sequence: runbook, changing active reviewer tracking. The flag principle holds in both directions: a save made under either flag value, with the recorded mode in either state, commits; statistics are served again once the flag returns to the recorded mode and the staled families are rebuilt, or once an owned M15 transition records the new mode.

Acceptance evidence

• Replica-set tests exercise assignment, screening assignment, annotation capacity save and screening capacity save with statistics enabled, and verify the durable read occurs inside a transaction.
• A real durable singleton disagreement (recorded untracked, this replica tracked) lets the guarded statistics screening save commit, and refuses the untracked overload for a caller that assumed the recorded mode, before changing the Study (#4286).
• A duplicate statistics operation rejected after the source replacement leaves the persisted Study, source revision and projection revision at their prior committed values.
• A screening save whose capacity/version guard does not match reaches no commit and leaves the stored document byte-identical, and an assignment that matched nothing reaches no commit either.
• A writes-flag refresh landing between the transaction decision and the agreement check cannot split them: the request that owned a transaction still validated the mode inside it.
• An assignment that matched nothing leaves a cached Study instance in place, while a committed one evicts it.
• Every statistics-writing transaction start is pinned to snapshot read concern by test, including the one SubmitAnnotationSessionService opens and lends to the annotation capacity save.
• A branch that consumes the durable mode follows the value its transaction verified, not a runtime flag refresh that arrives afterwards.
• Assignment recovers from a transient abort inside its budget and answers null when the budget is spent; a mutating callback runs exactly once.
• The annotation statistics save answers false on a stale version and on a transient abort; under a durable-mode disagreement it proceeds to its statistics half on this replica's mode (#4286).
• Through the real annotation owner: the issue's sequence (recorded mode untracked, tracking switched on while writes are off, then writes on) saves in both phases and stops serving the Fresh row; with writes on, an agreeing save serves Fresh and a disagreeing one saves and stales (#4286).
• A save whose caller assumed a mode the verified one no longer matches is capacity-guarded rather than committed unguarded — including when no statistics commit is prepared — and an unverifiable mode refuses before the study is mutated.
• Both untracked statistics saves refuse when the mode moves under them, and allow a save whose caller assumed the mode it actually finds.
• A deletion re-chooses its mutation when the mode moves under it, rather than committing a plain delete for a reviewer whose slot should have been restored.
• A stale replica cannot hand out review work without a reservation through the service or the hub, and neither reads a control row when materialized writes are off.
• A save with no delta still opens its transaction and compares the assumed mode while writes are on, and still uses the plain save when they are off.
• The direct-navigation grant reads the study in the snapshot that verified the mode, and answers null — sending the caller to the atomic claim — when that mode says tracking is available, and the controller decides whether to run it from that same verified mode.
• A random assignment whose mode turns on after its pre-loop lookup was skipped returns the existing reservation rather than claiming a second slot, in all three loops — annotation, screening and the combined stage.
• A refused untracked save is followed by a re-resolution on the next attempt, so the retry takes the guarded overload instead of exhausting the budget on a conflict.
• An indeterminate assignment commit evicts the cached Study; an indeterminate submission commit is retried rather than failing the submission.
• Domain events raised by a statistics save reach the dispatcher exactly once, and only after commit.
• Existing flags-off tests prove no durable-mode read, while API filter tests prove the exception maps to the typed 503 and unrelated failures do not; repository tests prove a disagreement follows the process flag (claims, capacity saves and the direct-navigation grant) instead of raising it.

Accepted costs

• Under ReadConcern.Snapshot a concurrent version bump now raises a write conflict where the session-less guard previously returned a precise AtCapacity / VersionStale / NotFound diagnosis. The conflict is flattened to VersionStale, and ReviewController.TrySaveScreeningAsync recovers the precise answer on its bounded reload loop, so a genuinely at-capacity study can burn retries before reporting AtCapacity.
• The transaction is opened on materializedProjectStatisticsWrites alone, not on the narrower screening family flag and project allowlist. Enabling the global flag therefore costs a transaction on every assignment and every screening capacity save, including projects with no delta to write. Latency and retry acceptance at ½/5/10 reviewers remain separate measured evidence (#3255).

Reservation-family materialization, transaction domain-event delivery and staging activation retain their separate programme boundaries. This change introduces no schedule, consumer cutover or flag activation.