Server-side realtime plan: change streams, SignalR and the subscription model (October 2026)

This plan answers two questions from Chris: is Rx.NET the right way to turn MongoDB change streams into SignalR notifications, and what should the server-side subscription model be? It maps the current pipeline, compares the options against current .NET 10 / ASP.NET Core 10 facts, recommends a target architecture, and fixes a hub contract v2 for the parallel client plan to consume. It is planning only; nothing starts until Chris approves and answers the decisions in §9. Code facts are from main at 8f73bcbc9 (4 Oct 2026). Markers: VERIFIED (code path read), CORRECTED (an earlier finding was wrong), NOT VERIFIED (reason given). The client rewrite is out of scope (see state-management plan §5b).

Chris decisions recorded 2026-10-04: - Protocol: the API-wide Newtonsoft → System.Text.Json migration is a prerequisite, planned separately in system-text-json-migration-plan-2026-10.md. Hub v2 uses the STJ hub protocol. - Full-stats push: add it back on v2 as a follow-up release (R4), staging only; production is not planned for now. The legacy SubscribeToProjectFullStats is still retired in R0.1 (D5).

Chris decisions recorded 2026-10-05: - One realtime model. Every realtime update comes from change-feed readers, including statistics invalidations. There is no message-bus path for SignalR delivery (§4.7). This replaces the "hybrid" trigger that D2 used to recommend. - No unnecessary stopgap work on messaging infrastructure. Code that the v2 pipeline or the Wolverine switch will replace is not fixed first, unless a production defect forces it. So the interim Rx fix for #3974 is dropped (§2 RT1/RT2, §8). Instead, the v2 pipeline first serves the existing v1 hub behind the server-only flag realtimePipelineV2 (§6), so production can leave Rx without waiting for hub v2. - Staging is disposable. This affects the STJ plan's digest concern (its D6), not this plan.

Baseline (re-stated 2026-10-05). Two code baselines are used: - Re-read on 2026-10-05 at af2d8e960 (origin/main, the base of PR #4074; the PR changes only docs, so its heads carry the same code): the claim-revocation row in §1.1; the RT1, RT2 and RT10 fix columns in §2; §4.2's v1 registry notes; the health facts in §4.6; §4.7; the R0.1 test-file list; R1.1's health criteria; R1.4; R1.5; R3.3; the R4 trigger; and the 2026-10-05 entries in §10. - Everything else is from 8f73bcbc9 (4 Oct 2026) and was not re-read. Line numbers there may have drifted by the commits between the two.

1. Current state

1.1 Pipeline map (VERIFIED unless marked)

Stage	How it works today	Evidence
Change streams	One cached hot stream per aggregate type per API pod, opened lazily on the first subscriber: Observable.Create loop over WatchAsync, FullDocument = UpdateLookup, op-type $match only (no projection, no per-project filter), then RetryWithExponentialBackoff(15, 2 s…180 s) → Publish().RefCount()	MongoContext.cs:206-314; MongoRepositoryBase.cs:38-39
Streams opened in practice	Project (project details + listings), DataExportJob, Study (presence or full stats only). Presence is off everywhere (activeReviewerTrackingEnabled default false; no cluster-gitops override), and full stats has no web caller (row RT3), so production most likely runs 2 streams per pod. NOT VERIFIED live (no metrics read)	env-mapping.yaml:1596-1598; FeatureFlags.cs:35-36; grep of cluster-gitops
Resume tokens	Stored per event after OnNext, in memory only (InMemoryResumePointRepository, MongoLamarRegistry.cs:50). They survive an in-process retry, not a pod restart. Invalid token → cleared, then OnError	MongoContext.cs:220-235,267-288
Per-subscriber fan-out	Every subscription builds its own Rx chain over the whole collection stream and filters by id in memory, with three LogInformation calls per event per subscriber	MongoRepositoryBase.cs:165-212
Subscription registry	ConnectionKeyedSubDictionaryCache → per type, per connection, per key (("ENTITY", id), "COLLECTION", ("FULL-STATS", p, u), ("STUDY-PRESENCE", p, s)) → IDisposable	AggregateRootEntitySubscriptionManager.cs:29-47,187-262,279-282,417-418
Subscriber retry	A second, unfiltered retry (MyUtils.cs:588-640, 24 h budget) wraps each subscriber chain. Two retry operators with different semantics coexist	AggregateRootEntitySubscriptionManager.cs:60-71
Delivery	_hubContext.Clients.Client(connectionId).X(dto) from inside the Rx callback. The legacy path (authority mode Off, the deployed default) maps DTOs inline on the change-stream loop, including a synchronous Investigators.GetMany per subscriber per event	NotificationHub.cs:1155-1190,1255-1296; ProjectStandardDto.cs:137-144
Groups	User Group: {id} (joined on connect), Project Group: {id} (joined on SubscribeToProject). Nothing is sent to either group on main; #3932 adds the first user-group send	NotificationHub.cs:92-104,1148,1194; gh pr diff 3932
Authorization at subscribe	Hub policies via SignalRAuthorizationHandler (arg 0 = project id, or job id resolved to project). Enforced mode (M4b) re-decides with RealtimeDeliveryAuthority and, for payload streams, again at delivery (late-query barrier)	SignalRAuthorizationHandler.cs:95-145; NotificationHub.cs:1394-1408; AggregateRootEntitySubscriptionManager.cs:476-537
Revocation	TakeUntil(ProjectAccessRevoked…) per stream, plus RevokeProjectStreams on a disabled-member update	AggregateRootEntitySubscriptionManager.cs:596-640; NotificationHub.cs:1159-1166,1212-1231
Disconnect	Unsubscribes project and study managers, then review-session cleanup (Mongo ReviewSessionConnection + Quartz timers). Not the DataExportJob manager	NotificationHub.cs:112-131,143-320
Non-change-stream pushes	FEAT-024 statistics: Mongo outbox → ProjectStatisticsDispatchService (1 s) → bus publish → per-pod temporary queue → per-connection authority recheck → invalidation-only ProjectStatisticsChanged. Claim revocation: same pattern → Clients.User. Search deletion (deletion lifecycle S4a): project-management publishes ISearchDeletedEvent → per-pod temporary queue api-search-deletions → per-subscriber View recheck over ProjectStatisticsSubscriptions → SearchDeleted (Deletion/SearchDeletedConsumer.cs). Flag revision: Clients.All on the handling pod only. #3932 (open): a raw WatchAsync loop on the inbox → InboxChanged() to the user group. Each moves to a change-feed reader under the one model (§4.7)	Statistics/*.cs; ClaimRevocations/*.cs; RuntimeFeatureFlagsController.cs:153. Claim-revocation path re-read 2026-10-05 (VERIFIED): intents written in the source transaction to pmActivityClaimRevocationOutbox (ActivityClaimRevocationOutbox.cs:18,49; writers StudyRepository.ActivityReservations.cs:93, StageReviewSettingsController.cs:303) → ActivityClaimRevocationDispatchService (1 s PeriodicTimer, :19) → ActivityClaimRevocationSink IBus.Publish (:10-13) → temporary queue api-activity-claim-revocations per pod (ActivityClaimRevokedConsumer.cs:92-100) → recheck → Clients.User (:32)
Scale-out	No backplane (no AddStackExchangeRedis for SignalR). Production API HPA 3–10 pods. Every pod opens its own streams; cookie session affinity is on every SyRF ingress	Program.cs:379-400; cluster-gitops production/api/values.yaml:46-50; _ingress.tpl:20-23
Reconnect	Server: no AllowStatefulReconnects. A reconnect gets a new connection ID and no subscriptions (client bug #3976). A pod restart loses everything; nothing replays	Program.cs:822; signal-r.service.ts:633

1.2 Hub surface (VERIFIED)

NotificationHub.cs is 1,650 lines with 15 client-callable methods: - 5 review-session commands: JoinStudyReview, LeaveStudyReview, StartedAnnotating, StoppedAnnotating, Heartbeat; - 10 subscription methods: subscribe/unsubscribe for StudyPresence, DataExportJob, Project, ProjectSummaries and ProjectFullStats.

Server→client messages:

Message	Trigger	Payload	Audience on main
ProjectNotification	Project change stream	Full ProjectWithRelatedInvestigatorsDto or delete	Subscriber of that project
ProjectSummariesNotification	Every Project write in the database	Summary DTO + investigators, or delete when not visible	Every listing subscriber, mapped per connection
DataExportJobNotification	DataExportJob change stream	DataExportJobDto	Job subscriber
ProjectStatsNotification	Every Study write in the project, or a Project write	Per-investigator FullStats	Full-stats subscriber (none in the web)
StudyPresenceUpdated	Study change stream (presence fields distinct)	StudyReviewPresenceSnapshot	Presence subscriber (flag off)
ProjectStatisticsChanged	FEAT-024 outbox → bus	Revision strings (invalidation)	FEAT-024 registry
ActivityClaimRevoked	Claim-revocation outbox → bus	Own claim DTO	Clients.User
RuntimeFeatureFlagsRevision	Admin PUT	long revision (invalidation)	Clients.All, local pod only
UiVersionCheck	On connect	Min UI version	Caller
InboxChanged (#3932, open)	Inbox change stream	Empty (invalidation)	User group

2. Scope

ID	Problem	Impact	Evidence (file:line)	Issue
RT1	Subscribers run inline on the cursor loop; a throwing subscriber reaches the loop's catch, and shouldRetry rejects non-Mongo exceptions	Live updates stop on that pod. Mechanism VERIFIED; whether Publish() then stays terminated for good is NOT VERIFIED and is no longer worth proving, because the Rx path is replaced rather than fixed	MongoContext.cs:262-265,289-306	#3974. Fix: R1.1 + R1.2 under realtimePipelineV2 (readers run no subscriber code; every send is isolated). The interim Rx fix (old backend PR-5 change-stream part) is dropped as stopgap work (Chris, 2026-10-05)
RT2	Health records success only on an event, the retry budget is cumulative, and the check is tagged live	Liveness kills pods after the database recovers; after 15 failures with no event in between, the stream is gone	MongoContext.cs:302-310,397-405; ChangeStreamHealthTracker.cs:57-82; Program.cs:167-170	#3974. Fix: R1.1 (readers retry without a cap and record health when the cursor opens). Change-feed health is not part of readiness (Chris, 2026-10-05): a feed problem reports Degraded plus a metric and alert, and only a gated liveness check restarts a pod whose reader is stuck while the database answers (§4.6). Until realtimePipelineV2 is on for a host, the Rx change-streams check stays as it is today (no stopgap work)
RT3	CORRECTED. The review says every Study save re-runs the full-stats aggregate for every viewer. The server path is real: CombineLatest → sync Projects.Get + GetFullStatsForInvestigatorAsync per emission, no throttle, unbounded .Merge() (out of order), DateTime.Now. But the web has not invoked SubscribeToProjectFullStats since 2026-04-17 (commit 0cea246dc7: "no client dispatches it"). It is dead code that any member can still call, so it is a cost amplifier	Latent DoS vector, ~400 lines of server code and tests	AggregateRootEntitySubscriptionManager.cs:301-367,476-537 (:332,340,348,507,516,523); NotificationHub.cs:1305-1349; web grep returns 0 callers	new
RT4	Head-of-line blocking: the legacy DTO mapping does a sync GetMany per subscriber per event on the cursor thread	One slow read delays every subscriber of that collection on the pod	NotificationHub.cs:1178,1285; ProjectStandardDto.cs:140	new
RT5	Listing fan-out: every Project write in the database is mapped once per listing viewer	Cost grows with writes × viewers × pods	NotificationHub.cs:1236-1303	new
RT6	Full 118–465 KB Project documents (UpdateLookup) reach every pod on every Project write	Network and deserialisation on 3–10 pods	MongoContext.cs:221-225; size from repository-cache.md	new
RT7	Log volume: 3 Info lines per event per subscriber	Log cost; noise	MongoRepositoryBase.cs:169-212	new
RT8	Export-job subscriptions are not removed on disconnect	Leak per connection	NotificationHub.cs:112-122	#3978 (backend PR-9)
RT9	Reconnect loses subscriptions; no missed-event strategy; DbTime has one-second resolution and no DTO carries Audit.Version	Silent staleness; clients cannot order two changes inside one second	MongoUtils.cs:18-21; Models/ProjectDto/*.cs (no version field)	#3976 (client)
RT10	Pushes not driven by change streams reach only the local pod (Clients.All for flag revisions)	Other pods' clients wait for the 5-minute poll (non-production only)	RuntimeFeatureFlagsController.cs:153	new; fixed by the R1.5 reader on runtimeFeatureFlagOverrides (§4.7)
RT11	Three change-stream idioms: Rx cached stream, the dead blocking variant, and #3932's raw loop with no resume or health	Each fixes the same problems differently	MongoContext.cs:173-201; #3932 InboxInvalidationWorker.cs	new
RT12	The hub mixes review-session commands, 10 subscription methods and two authority modes in 1,650 lines	Hard to change safely; four open PRs touch it	NotificationHub.cs	new
RT13	SignalR detailed errors are always on	Exception text reaches clients	Program.cs:381	#3978 (backend PR-8)

3. Options evaluated (web facts checked 2026-10-04)

Option	Facts	Verdict
Keep Rx.NET, cleaned up	Rx.NET is maintained: 6.0.2 (2025-08-29), 6.1.0 (2025-10-03), 7.0.0 (2026-07-17), targeting net8.0/netstandard2.0 (NuGet). SyRF pins 6.0.0 (SyRF.Mongo.Common.csproj:20, SyRF.SharedKernel.csproj:36) and uses it in 4 production files.	Viable, but the defects RT1/RT3/RT4 come from Rx idioms (hot shared subjects, inline observers, FromAsync+Merge, two retry operators) that few maintainers can reason about. Cleaning up keeps the per-subscriber pipeline model (RT5/RT6). Not recommended as the target. No interim Rx fix either (Chris, 2026-10-05): the old #3974 stopgap is dropped, and realtimePipelineV2 replaces Rx behind v1 instead
BackgroundService + System.Threading.Channels + IAsyncEnumerable	All in-box. System.Linq.AsyncEnumerable is in the .NET 10 shared framework (VERIFIED: ~/.dotnet/shared/Microsoft.NETCore.App/10.0.0/System.Linq.AsyncEnumerable.dll; package 10.0.0 2025-11-11). The FEAT-024 and claim-revocation dispatchers and #3932's worker already use BackgroundService loops. TimeProvider makes windows testable	Recommended. Straight-line await foreach with try/catch, one reader per collection, explicit bounded queues, per-send isolation
SignalR server-to-client streaming (IAsyncEnumerable/ChannelReader hub methods)	One producer per invocation; it ends on disconnect and must be re-invoked; no group fan-out	Not adopted: it re-creates per-connection pipelines
Notify-then-fetch (invalidation)	Already the house pattern for FEAT-024 (ProjectStatisticsChanged.cs:5-10), flag revisions (feature-flags.md) and #3932. Fits the client plan's resources (reload on event)	Recommended for project details and listings; small data payloads stay where a refetch would cost more than the payload (§4.4)
Stateful reconnect	.NET 8+: server AllowStatefulReconnects on MapHub, StatefulReconnectBufferSize default 100,000 bytes; client withStatefulReconnect() (docs). The buffer lives in the pod, so it needs affinity, and a pod restart still loses it. Protocol support VERIFIED (orchestrator, 2026-10-04): dotnet/aspnetcore main NewtonsoftJsonHubProtocol.cs declares ProtocolVersion = 2 and handles AckMessage/SequenceMessage, so both JSON protocols support it. WebSockets-only is reported by secondary sources; NOT VERIFIED in primary docs	Recommended on v2, as a latency optimisation, not as the correctness mechanism (versions + resync are)
Domain events via the outbox instead of change streams	The outbox is persistence-plan PR-10 (flagged, needs #3973 and PR-5). Events cover only unit-of-work saves: partial updates, raw writes, PM and Quartz writes bypass them. Wolverine switch is targeted for Q3 2027 (ADR-021)	Not adopted as a realtime trigger. DECIDED (Chris, 2026-10-05): change streams are the only realtime trigger. Entity invalidation comes from readers on the entity collections, because they see every committed write. Semantic events keep their durability without a bus hop: the source transaction already writes a durable "notification wanted" record (the FEAT-024 notification slot, the claim-revocation intent), and a reader watches that collection (§4.7). The earlier "hybrid" verdict (outbox → bus fan-out for statistics and claim revocation) is superseded
Wolverine SignalR transport	WolverineFx.SignalR 6.45.0 (2026-10-02) exists, but hubs must inherit WolverineHub and messages use a CloudEvents envelope (docs)	Not adopted: incompatible with the typed hub and its client. After S1, Wolverine handlers call the same IRealtimeClients facade
Backplane: none / Redis-Valkey / Azure SignalR	Redis backplane (docs) forwards group/user/all sends across servers; sticky sessions are still required. valkey-production exists for the (disabled) BFF. Azure SignalR doesn't apply on GKE	None. Under the one model every source is a change feed, and every pod opens its own readers, so every source reaches every pod. A backplane would add a hot-path dependency for one gap (RT10), which a per-pod reader on the flag-override collection closes (§4.7). Revisit triggers in D4

4. Target architecture

flowchart LR
  subgraph pod[Each API pod]
    R1[ChangeFeedReader<Project>] --> Q[(bounded Channel<EntityChange>)]
    R2[ChangeFeedReader<Study>] --> Q
    R3[ChangeFeedReader<DataExportJob>] --> Q
    R4[ChangeFeedReader<InboxNotification>] --> Q
    S[readers on the statistics slot collections, R1.4] --> SD[statistics delivery: same checks as today]
    CR[reader on claim-revocation intents, R1.5] --> CD[claim delivery: same recheck as today]
    FF[reader on runtimeFeatureFlagOverrides, R1.5] --> C
    SD --> C
    CD --> C
    Q --> D[RealtimeDispatcher: coalesce, revoke, authorise]
    D --> C[IRealtimeClients facade]
    REG[(RealtimeRegistry: topic → connections, v1 and v2)] <--> D
    V1[NotificationHub /notifications, v1 methods] --> REG
    H[RealtimeHub /notifications/v2, later] --> REG
    C --> V1
    C --> H
  end

There is no bus consumer in the realtime path (§4.7). Under realtimePipelineV2 the v1 hub's subscription methods register into the registry, and the dispatcher sends today's v1 messages. The v2 hub is a later step for contract changes only.

4.1 Change feed readers

• One ChangeFeedReader<T> BackgroundService per watched collection per pod, started at host start when realtimePipelineV2 is on (so health is meaningful) and running regardless of subscribers.
• Pipeline: op-type $match plus a $project that keeps only the routing fields: _id, ProjectId, Audit.Version, plus for Study the fields StudyReviewPresenceSnapshot.FromStudy reads, and for DataExportJob its status fields. Saves are replaces (MongoExtensions ReplaceOneAsync), which always carry fullDocument, so the projection trims them. UpdateLookup stays only for partial update ops.
• Loop: await foreach over the cursor, TryWrite into a bounded channel. Retry with uncapped backoff (2 s → 180 s ceiling, jitter). Healthy when the cursor opens; Degraded while retrying. Each reader records when it last had an open cursor, which the gated liveness check reads (§4.6).
• Resume token kept in memory.
• An invalid or lost token, or any restart without a token, emits a ResyncRequired for that collection, because events may have been missed.
• No subscriber code runs on the reader thread. A full channel coalesces rather than blocking the cursor.

4.2 Registry and dispatcher

• RealtimeRegistry (singleton): topic → connectionId → Subscription(subject, impersonationActor, hubKind, subscribedAt), plus a reverse index per connection for disconnect cleanup and per-user lookup. SignalR groups are not needed without a backplane.
• It serves v1 as well as v2 (re-stated 2026-10-05). Under realtimePipelineV2, today's v1 hub methods (SubscribeToProject, SubscribeToProjectSummaries, SubscribeToDataExportJob, SubscribeToStudyPresence and their unsubscribes) keep their names, policies and return values, but register into the registry (hubKind = v1) instead of the Rx AggregateRootEntitySubscriptionManager. OnDisconnectedAsync drops every registry entry for the connection, export jobs included (RT8, CR-4).
• With the flag off, the v1 methods still use the Rx managers, unchanged.
• The v2 hub (R1.3b) registers into the same registry with hubKind = v2.
• RealtimeDispatcher (one BackgroundService reading the channel):
• Map each change to topic keys.
• Coalesce per topic, latest wins, with a trailing window (§4.5).
• Skip topics with no local subscriber.
• For a Project change with local subscribers, run one uncached Project read per window per pod (and one investigator read for the payload) and evaluate revocation for every subscriber in memory (RevokedBy/IsDisabled, which exist today). Drop revoked subscriptions and tell the client in its contract's form: today's v1 revocation behaviour for v1 connections, and SubscriptionRevoked for v2.
• Build the message per hub kind. v1 connections get today's v1 messages with the same names and payloads (ProjectNotification, ProjectSummariesNotification, DataExportJobNotification, StudyPresenceUpdated), mapped per subscriber from the one read, off the cursor thread, with the legacy listing visibility predicate (NotificationHub.cs:1260-1267) and, under enforced mode, today's per-delivery authority for payload streams (M4b). v2 connections get the §5.2 messages.
• Send through IRealtimeClients. Each send has its own try/catch, and a failed socket never affects others.
• IRealtimeClients facade: Connection(id), Connections(ids), User(userId), All(). It fans out to both hub contexts (v1 NotificationHub now, v2 RealtimeHub once R1.3b lands). Every producer is a reader (§4.7): the statistics readers (R1.4), the claim-revocation and flag-override readers (R1.5) and the #3932 inbox reader all send through it.

4.3 Authorization

Moment	Rule
Subscribe	Unchanged policies on typed hub methods (SignalRAuthorizationHandler); under enforced mode, RequireCurrentAuthorityAsync as today
Revocation	Driven by the Project change (4.2 step 4) and by subject suspension: deny on the next invocation (SubjectAdmissionHubFilter, exists), and D8: abort the subject's local connections on a suspension event
Delivery, invalidation messages	No per-delivery read. They carry ids and versions only, and the refetch is authorised over HTTP. Same stance as FEAT-024 and #3932
Delivery, payload messages (presence snapshot, export status, claim revocation)	Per-delivery decision kept under enforced mode (M4b parity), batched per event

4.4 Payload versus invalidation per message

v2 message	Kind	Why
ProjectChanged	Invalidation	Project DTO is large and per-user; the client reloads its route resource
ProjectListingChanged	Invalidation, filtered by visibility (public, active member, pending invite or join request) so private project ids never leak	Today's per-user mapping cost (RT5)
ExportJobChanged	Status data (small)	Progress ticks would double the load as refetches; the client fetches the full job on a terminal status (GET api/projects/{p}/data-export/{j} exists, DataExportController.cs:75)
StudyPresenceChanged	Data (existing snapshot)	Small and latency-sensitive; no presence GET exists
ProjectStatisticsChanged, ActivityClaimRevoked, InboxChanged, RuntimeFeatureFlagsRevision, UiVersionCheck	Unchanged	Already right-sized

4.5 Throttling, delivery semantics, reconnect

• Coalescing windows (PROPOSAL): Project 1 s, listing 2 s, export 1 s, presence 250 ms. Each send carries the newest version seen.
• Delivery is at-most-once per connection. Correctness comes from three rules:
• (a) every subscribe returns a SubscriptionAck carrying the current version where one exists;
• (b) the client compares versions and reloads on any doubt;
• © ResyncRequired after server-side loss.
• version is Audit.Version. It is advisory until persistence-plan #3985 makes direct writes bump it; until then clients treat any invalidation as "reload". at is the change's cluster time as a "t.i" string, for diagnostics and ordering inside one second.
• Reconnect:
• v2 allows stateful reconnect, so a blip on the same pod replays the buffer;
• a new connection (or a pod restart) re-declares its topics (client plan RealtimeStore), and the acks drive reloads;
• review sessions keep their own rejoin protocol.

4.6 Observability and health

• Meter SyRF.Api.Realtime with bounded tags only (collection, message, outcome; never ids):
• realtime.change_feed.events, .restarts{reason}, .lag_seconds (now − cluster time);
• realtime.channel.depth, realtime.coalesced;
• realtime.subscriptions{topic_kind}, realtime.sends{message,outcome};
• realtime.resyncs.
• Logging through [LoggerMessage], Debug per event, Information per reader state change.
• Change-feed health (DECIDED by Chris, 2026-10-05). Two checks, neither of which gates readiness:
• realtime-change-feeds: never part of readiness. It reports Healthy when every reader has an open cursor and Degraded while any reader is retrying. It is tagged neither ready nor live. ASP.NET Core maps Degraded to HTTP 200 by default, and SyRF sets no ResultStatusCodes override (grep of src, VERIFIED), so it cannot fail a probe in any case. A degraded feed shows as the metrics realtime.change_feed.restarts and realtime.change_feed.degraded_seconds{collection}, with an alert on degraded time over 5 min (PROPOSAL). A change-feed problem never takes the API out of service.

• change-feed-stalled: a separate, gated liveness check (tagged live). It fails only when both are true:

  1. a reader has been continuously unable to open its cursor for at least StalledRestartAfter (default 15 min, PROPOSAL; configuration, not a flag); and
  2. a database ping (short timeout, through the same MongoClient) succeeds now.

  So a pod with a stuck reader restarts itself, but a database outage fails the ping and restarts nobody. Without the gate, a database outage would restart the whole fleet. Opening the cursor resets the timer. The API registers no Mongo ping check today (grep, VERIFIED), so the check issues its own ping. It is registered only when realtimePipelineV2 is on. - Why the old ordering caution is moot. An earlier draft moved change-stream health to readiness, which made re-tagging the Rx check ready before reader recovery unsafe (a dead stream on every pod would take all pods out of service). Readiness is now never gated by change feeds, so that trap no longer exists. With the flag on, the host opens no Rx stream and does not register the Rx change-streams check (Program.cs:167-170). With the flag off, that check stays exactly as today until R3.1 deletes it. - Backend PR-5's probe work that does not touch change streams (a live predicate on /health/live, Quartz SQL to ready, probe timeouts) is independent and stays in that plan.

4.7 One realtime model: every source is a change feed (Chris, 2026-10-05)

Every realtime update is triggered by a change-feed reader on the collection that records it. No bus consumer delivers to SignalR.

Source	Collection watched	How it reaches clients	Release
Project, listings, export jobs, presence	pmProject, pmStudy, the DataExportJob collection	Entity readers → dispatcher → v1 messages (and v2 later)	R1.1, R1.2
FEAT-024 statistics invalidations	pmProjectStatisticsNotificationOutbox (one coalescing slot per project + kind + family) and pmProjectStatisticsGlobalNotificationOutbox (mode and kill-switch transitions)	With realtimePipelineV2 on: slot readers → the same per-subscriber checks as today → ProjectStatisticsChanged, unchanged. With it off: today's bus path, unchanged (deleted in R3.3)	R1.4
Activity-claim revocation	pmActivityClaimRevocationOutbox (intents written in the source transaction)	With the flag on: intent reader on every pod → today's recheck → ActivityClaimRevoked to the owner's local connections. With it off: today's bus path (deleted in R3.3). PROPOSAL, pending the review-eligibility owner (the stream is paused)	R1.5
Runtime flag revision	runtimeFeatureFlagOverrides (RuntimeFeatureFlagStore.cs:71, VERIFIED)	Reader on every pod → apply the snapshot locally, then RuntimeFeatureFlagsRevision to local connections. This closes RT10 (today only the handling pod broadcasts, RuntimeFeatureFlagsController.cs:153, and other pods wait for the 30 s refresh, RuntimeFeatureFlagProvider.cs:172)	R1.5
Inbox (#3932)	The inbox collection	Already a per-pod change stream in #3932; moves onto ChangeFeedReader (resume, health)	Follow-up with the notifications stream

• Why the statistics trigger is the slot collection, not the control row. The control row's ClientInvalidationRevision is one project-wide counter (ProjectStatisticsControl.cs:71, bumped at :197 and :510, VERIFIED). It cannot say which family changed. ProjectStatisticsChanged carries NotificationKind and MetricFamily (SignalR/Statistics/ProjectStatisticsChanged.cs:7-10, VERIFIED), and the web refetches only for a family it subscribes to, or on a visibility or global event (project-statistics-invalidation.service.ts:69-82, VERIFIED). The slots carry kind and family (ProjectStatisticsNotificationOutbox.cs:41-52, VERIFIED).
• The durable records stay. The slot writes and claim-revocation intents are written in the source transaction, as today. Only the delivery side changes. Exactly one path is active per host, chosen by realtimePipelineV2; the old path (dispatcher, lease, sink, bus, temporary queue) is deleted only in R3.3 (R1.4 item 5).
• Recovery without a bus. A reader that restarts without a resume token cannot know what it missed. The slot and intent collections hold durable state, so the reader catches up by reading them (§7 R1.4 item 4). For the entity readers, ResyncRequired serves v2 connections; v1 has no such message, so a v1 subscriber gets a fresh v1 message for its topic (one read per topic per pod).

5. Hub contract v2 (for the client plan)

Endpoint: RealtimeHub : Hub<IRealtimeClient> mapped at /notifications/v2 with AllowStatefulReconnects = true. This path reuses the existing ingress prefixes (staging's Web-host /notifications rule) and query-token auth (DirectBearerSchemeSelector.cs:82 uses StartsWithSegments("/notifications")), both VERIFIED. It also doesn't collide with MapHub("/notifications") (to be proved by an R1.3 test).

Protocol (decided by Chris, 2026-10-04): the System.Text.Json hub protocol (AddJsonProtocol), with serializer options identical to the API's migrated MVC options (camelCase, the same enum, null and converter policy). This makes the API-wide Newtonsoft → STJ migration (system-text-json-migration-plan-2026-10.md) a prerequisite of R1.3b. - Today the hub uses AddNewtonsoftJsonProtocol with three custom converters and a camelCase resolver (Program.cs:390-400), VERIFIED. - Both protocols register under the protocol name json, so one host cannot offer Newtonsoft on v1 and STJ on v2 side by side. VERIFIED 2026-10-04 against dotnet/aspnetcore main: NewtonsoftJsonHubProtocol.cs:39 and JsonHubProtocol.cs:49 both declare ProtocolName = "json". The migration therefore switches the v1 hub's protocol too, as part of the same API-wide change. - Every v2 payload type (StudyReviewPresenceSnapshot, StudyReviewAccessResult, ActivityClaimRevokedDto, ProjectStatisticsChanged, and the R4 ProjectStatisticsSnapshot sections) gets an STJ round-trip test, either in the STJ plan or in R1.3b, whichever lands first. - Stateful reconnect works on either protocol (§3, VERIFIED), so the protocol choice does not gate D7.

Versioning: - contractVersion = 2 is returned in every ack. - Additive changes only within v2 (new messages, new optional fields). Clients ignore unknown messages and fields. - Any rename, removal or type change means /notifications/v3. - The contract is pinned by a .NET snapshot test of method signatures and message JSON, committed as docs/architecture/realtime-contract-v2.md plus a JSON sample file that a web spec reads (D9).

5.1 Client → server methods

Method	Policy	Returns	Notes
SubscribeProject(Guid projectId)	ProjectViewSignalRPolicy	SubscriptionAck	Also registers the FEAT-024 statistics subscription (filter updated to the v2 name)
UnsubscribeProject(Guid projectId)	none	void	Idempotent; never throws
SubscribeProjectListing()	authenticated	SubscriptionAck	One per connection
UnsubscribeProjectListing()	none	void	Idempotent
SubscribeExportJob(Guid exportJobId)	ProjectExportDataSignalRPolicy (job → project)	SubscriptionAck with the current status
UnsubscribeExportJob(Guid exportJobId)	none	void	Idempotent
SubscribeStudyPresence(Guid projectId, Guid stageId, Guid studyId)	ProjectViewStudiesSignalRPolicy	PresenceAck (ack + current snapshot)	No-op ack when tracking is unavailable
UnsubscribeStudyPresence(Guid projectId, Guid stageId, Guid studyId)	none (today it needs View, NotificationHub.cs:1076)	void	Idempotent
SubscribeProjectStatistics(Guid projectId, Guid? stageId) (R4, additive)	ProjectViewStudiesSignalRPolicy (member content, as the legacy full stats), plus the per-delivery member-content check in every mode (R4 item 7)	SubscriptionAck (+ a snapshot read fresh for this caller when served)	Denied with reason: "featureUnavailable" (an additive value on top of the CR-3 reasons) when realtimeStatisticsPush is off or the project is not allowlisted. stageId selects the per-stage (Stage Overview) read
UnsubscribeProjectStatistics(Guid projectId, Guid? stageId) (R4)	none	void	Idempotent
JoinStudyReview, LeaveStudyReview, StartedAnnotating, StoppedAnnotating, Heartbeat	Unchanged	Unchanged (StudyReviewAccessResult, bool)	Same signatures; both hubs delegate to one extracted ReviewSessionCommands service

Not carried over: SubscribeToProjectFullStats (RT3); R4's SubscribeProjectStatistics replaces it on a different design. Groups are not part of the contract; topics are server-internal (project:{id}, listing, export:{id}, presence:{p}:{s}, user:{id}).

5.2 Server → client messages

Message	Shape (camelCase)	Kind
ProjectChanged	{ projectId, change: "updated" \| "deleted", version?: number, at: string }	Invalidation
ProjectListingChanged	{ projectId, change: "updated" \| "removed", version?: number, at: string }	Invalidation
ExportJobChanged	{ exportJobId, projectId, version?: number, status: string, progress?: number, terminal: boolean, at: string }; the exact status fields are fixed in R1.3 from DataExportJobDto	Data (status)
StudyPresenceChanged	{ projectId, studyId, version: number, snapshot: StudyReviewPresenceSnapshot }	Data
SubscriptionRevoked	{ topic: "project" \| "listing" \| "export" \| "presence", id?: string, reason: "accessRevoked" \| "deleted" }	Control
ResyncRequired	{ scope: "all" \| "project" \| "listing" \| "export" \| "presence", id?: string, reason: "feedRestarted" \| "resumeLost" }	Control
ProjectStatisticsChanged	Unchanged (ProjectStatisticsChanged.cs)	Invalidation (FEAT-024-owned)
ActivityClaimRevoked	Unchanged	Data
InboxChanged	Unchanged from #3932 (no arguments)	Invalidation
RuntimeFeatureFlagsRevision	Unchanged (long)	Invalidation
UiVersionCheck	Unchanged (string)	Control
ProjectStatisticsSnapshot (R4, additive)	{ projectId, stageId?: string, revision: string, globalRevision: string, sections: { projectScreening?, stageAnnotation?, membershipScreening?, membershipAnnotation? }, at: string }. Each section is the HTTP bundle section for this caller (byte-equal, R4 4.1.3) plus provenance { servedFrom: "materialized", checkpointId, pendingFingerprint, watermarks }. revision is the Int64 project ClientInvalidationRevision and globalRevision the Int64 GlobalClientInvalidationRevision, both read in the same snapshot, sent as strings and compared as integers (the client's pollCoherentStatistics tracks both). A section is present only when its bundle is Fresh and the HTTP consumer flag for it is on, and only for sections backed by the two coherent DTOs (StageOverviewStatistics, ScreeningOverviewStatistics; R4 item 10). Exact field names are fixed in R4.1 with the statistics session	Data (shaped per subscriber, as the materialised HTTP read)

SubscriptionAck = { contractVersion: 2, topic, id?: string, version?: number, at: string }.

Client obligations: subscribe before fetching, or compare the ack version with the fetched one. Re-declare every topic on a new connection ID. Reload on ResyncRequired. Never treat an absent message as "unchanged".

5.3 Reconciliation with the client plan (orchestrator, 2026-10-04)

Each client requirement (CR-n in realtime-client-plan-2026-10.md §9) is resolved below. These resolutions are part of contract v2 from its first release.

CR	Resolution
CR-1 invalidation shape	Adopt the server's per-family messages (ProjectChanged, ProjectListingChanged) rather than a generic { entityType, … }. The client's message typing maps families to resources.
CR-2 per-subscription seq	Deferred (additive later). Gaps are covered by stateful reconnect, ResyncRequired (feed restart or lost resume token), refetch-on-new-connection and the version > check. If staging shows missed updates, add an optional seq to v2 messages; this is additive. The client's RV.2 is deferred with it.
CR-3 ack or denial as a return value	Adopt. SubscriptionAck gains granted: boolean, reason?: "notAMember" \| "notFound" \| "disabled" \| "authorityUnavailable" and retryable: boolean. Subscribe methods return a denial instead of throwing a HubException, and policy failures are mapped to reason. Turning off EnableDetailedErrors in production (backend plan #3978) is independent.
CR-4 batch resubscribe and full cleanup	Adopt. Add Resubscribe(SubscriptionRequest[] requests) → SubscriptionAck[] (SubscriptionRequest = { topic, id?, stageId?, studyId? }). The registry drops every topic for a connection on disconnect, including export jobs; R1.2 gets an acceptance criterion for it.
CR-5 revocation notice	Covered by SubscriptionRevoked.
CR-6 stateful reconnect	Covered (AllowStatefulReconnects = true). Closed, 2026-10-04: NewtonsoftJsonHubProtocol declares ProtocolVersion = 2 and handles AckMessage/SequenceMessage (VERIFIED by the orchestrator in dotnet/aspnetcore main), so stateful reconnect does not depend on the protocol. v2 nonetheless uses the STJ protocol by Chris's decision (§5 "Protocol"). R1.3b keeps one stateful-reconnect test on STJ.
CR-7 delete versions	ProjectChanged { change: "deleted" } carries version when known. It stays advisory until #3985 makes every write bump Audit.Version.
CR-8 contract manifest	Covered by the snapshot test plus docs/architecture/realtime-contract-v2.md and the JSON sample the web spec reads. contractVersion arrives in every ack, so no separate on-connect message is needed.
CR-9 ordered delivery	Covered: one dispatcher per pod and per-topic coalescing; the full-stats .Merge() path is removed in R0. The client still applies version > checks.
CR-10 user-group delivery without a backplane	Covered by D4 provided the source reaches every pod. #3932's InboxChanged reader must run on every API pod (it does today as a per-pod change stream). Any future user-group message sent from a single consumer needs a fan-out endpoint, like ProjectStatisticsChanged.
CR-11 keep-alive values	The server uses the ASP.NET Core defaults today (AddSignalR sets no intervals: Program.cs:379-389): keep-alive 15 s, client timeout 30 s. The contract document states them, and the client sets withKeepAliveInterval(15 s) and withServerTimeout(30 s). Changing them later is a contract change on both sides.

Flags (revised 2026-10-05; §6 has the full decision): - realtimePipelineV2 (server only, api) replaces Rx behind today's v1 hub. It needs no client change, because v1 message names and payloads stay the same. - realtimeHubV2 (api + web) adds the /notifications/v2 contract later. It requires realtimePipelineV2 and stjHubProtocol. - realtimeClientV2 (web, page reload) selects the new client core. It runs on today's v1 hub, so it does not depend on either server flag. CORRECTED (2026-10-05): this section used to say it requires realtimeHubV2. That conflicted with the client plan, whose R1/R2 run on today's hub. Only the client's R3 adoption of contract v2 (RV.x) needs realtimeHubV2.

Rollout order per environment: realtimePipelineV2 on, then the parity checks (R2); realtimeHubV2 later, after R1.3b. Rollback is the reverse.

6. MVP, flag decision and common criteria

MVP boundary (revised 2026-10-05): R0 + R1.1 + R1.2 + R2. The v2 pipeline (readers, registry, dispatcher) serves today's v1 hub behind realtimePipelineV2, with the same message names and payloads, proven by hermetic E2E and then on staging. That lets production leave Rx (a Chris-approved step) without waiting for hub v2. After the MVP: R1.3a/R1.3b (hub contract v2), R1.4 (statistics via change feed, timed by the statistics programme), R1.5 (claim revocation and flag revision), R3 (Rx deletion, then v1 retirement) and R4. Out of scope (tracked where noted): - client code (state-management plan §5b and its follow-up client plan); - an interim Rx fix for #3974: dropped as stopgap work (Chris, 2026-10-05). R1.1 replaces it. Backend PR-5 keeps only its change-stream-independent probe work; - export-disconnect leak and detailed errors (backend PR-9/PR-8, #3978); - outbox (persistence PR-10); - Wolverine (ADR-021); - the API-wide Newtonsoft → STJ migration (a prerequisite, planned in system-text-json-migration-plan-2026-10.md); MessagePack is not planned; - R4 (full-stats push, staging only) is a follow-up release outside the MVP.

Cross-pod flag revisions (RT10) are now in scope as R1.5 (a reader on the override collection, not a bus fan-out).

Flag decision: flagged. The change is a hot-path rewrite with changed observable semantics, so it needs an environment-scoped kill switch. Two catalogued flags, both default false, via env-mapping.yaml and pnpm run generate:flags: - realtimePipelineV2 (services api only; no web tsName). It is the switch R1.1, R1.2, R1.4 and R1.5 sit behind. This name is already used in the implementation PR (orchestrator, 2026-10-05; NOT VERIFIED here, because no local worktree contains it). - The API reads the deployed value at start-up. With it on, the host starts the readers, registry and dispatcher, routes the v1 hub's subscription methods into the registry, and opens no Rx stream. No runtime override starts or stops them, as with the fold flag. - With it off, the v1 hub runs on Rx exactly as today. - The web sees no difference: same endpoint, same v1 messages. - Rollback is a values flip plus a pod restart. - realtimeHubV2 (services api + web tsName; later). It maps /notifications/v2 (R1.3b). Its catalogue entry requires realtimePipelineV2 and stjHubProtocol. The web uses the runtime value to choose /notifications/v2 or /notifications; only the new client core (client plan R3) honours it. - R0 is not flagged: it removes dead or broken behaviour with no web caller (D5). - R4 sub-flag realtimeStatisticsPush (api + web, default false). Its catalogue entry requires realtimeHubV2, materializedProjectStatisticsSignalR, materializedProjectStatisticsServing and the per-section consumer flags the HTTP endpoints check (for example …StageOverviewBundle, …ScreeningInfo, …ProjectOverview); R4 also checks those flags at run time on every pass (statistics session review, 2026-10-05). Why a sub-flag rather than reusing realtimeHubV2: - it has a different owner (FEAT-024) and a different risk (a payload carrying reviewer data); - it is staging-only while realtimeHubV2 is meant to reach production; - it needs its own kill switch so the statistics session can turn it off without taking the v2 hub down.

Production values stay false, and enabling it in production is outside this plan.

#	Common acceptance criterion (every PR)	Verification
C1	With realtimePipelineV2 off, v1 behaviour is unchanged: existing SyRF.API.Endpoint.Tests/SignalR/* pass untranslated (except files R0/R3 delete)	dotnet test --filter FullyQualifiedName~SignalR
C1b	With realtimePipelineV2 on, every v1 message keeps its name and JSON payload: the behaviours those tests assert (delivery, revocation, enforced-mode delivery authority, listing visibility) hold against the registry path	The same tests, parameterised on the flag; a payload snapshot per v1 message compared across both values
C2	No subscriber or send can throw into a reader or the dispatcher loop	Unit test with a throwing fake sender
C3	Metrics carry only bounded tags; no project, user or connection id	Meter tag-allowlist test (pattern from ProjectStatisticsMetrics)
C4	New code is in BackgroundService/Channel/TimeProvider idioms; no new System.Reactive use	Review; grep in the PR
C5	Docs in the same PR: docs/architecture/realtime-contract-v2.md and docs/features/signalr-active-reviewer-tracking.md where touched; flag decision stated	Docs review
C6	Narrow, niced test runs only (this machine is the CI host); settled review on the head; green checks	PR body; pr-review-settled.sh

7. Releases → PRs

R0: remove dead and hot-path waste (parallel; not flagged)

PR-R0.1 Retire the full-stats push (RT3; effort S–M). The statistics session confirmed on 2026-10-05 that there is no web caller and that FEAT-024 does not depend on this path (ProjectStatisticsSubscriptionFilter records only SubscribeToProject/UnsubscribeFromProject; the outbox, ProjectStatisticsDispatchService and the sink never touch AggregateRootEntitySubscriptionManager). MembershipScreeningVisibilityShaper survives (it is still used by ProjectController.cs:1187-1190, ReviewController.cs:1205-1210, ReviewerScreeningHistoryController.cs:70 and StageOverviewStatisticsQuery.cs:220,324), and so does GetFullStatsForInvestigatorAsync (ProjectController.cs:1182). The file list below includes the gaps that review found (statistics session review, 2026-10-05). Files: - NotificationHub.cs:1305-1349; - AggregateRootEntitySubscriptionManager.cs:279-410,463-537 (the full-stats parts only); - INotificationHubClient.ProjectStatsNotification; - RealtimeSurface.FullStatistics (SignalR/Authority/RealtimeDeliveryAuthority.cs:50), which has no other production use after R0.1; - the dead blocking stream GetCollectionChangeStream with its only caller MongoChangeStream (MongoContext.cs:173-201, MongoRepositoryBase.cs:41-46), and the unused DebugSubscribe/DebugSubscribeObservable (MongoContext.cs:366-370,435-461); 0 callers each (grep).

Tests to delete or translate (every reference to the full-stats method, message or surface): - SignalR/FullStatsPushVisibilityTests.cs (deleted, after the port below); - SignalR/NotificationHubCurrentAuthorityTests.cs (153-294, 373); - SignalR/NotificationHubDisabledMembershipRevocationTests.cs (137-249, 385); - Authorization/LegacyStatisticsAuthorizationTests.cs:187-191,228-229; - Authorization/EndpointAuthorizationCatalogTests.cs:210; - Authorization/EndpointAuthorizationExceptions.cs:269; - ApiSensitiveLogTests.cs:345-348; - SignalR/RealtimeDeliveryAuthorityTests.cs:232; - SignalR/AggregateRootEntitySubscriptionManagerLoggingTests.cs:82-88 (it tests LogFullStatsRetriesExhausted, which R0.1 removes with the full-stats parts at AggregateRootEntitySubscriptionManager.cs:366,370,536). Added 2026-10-05.

CORRECTED count (2026-10-05): these are 9 files, not 7. The list above had 8 entries. A grep of SyRF.API.Endpoint.Tests for FullStats, SubscribeToProjectFullStats, ProjectStatsNotification and FullStatistics finds the logging test as the ninth (VERIFIED). EndpointAuthorizationExceptions.cs is a test-support file, not a test class, but it names UnsubscribeFromProjectFullStats, so it counts. Three other matches (ProjectScreeningEndpointTests, ReviewControllerFullStatsMaterializedTests, ScreeningLeaderboardVisibilityEndpointTests) test the HTTP FullStats read, which survives, so they are not affected.

Visibility-test port inside R0.1. The shaper rules that FullStatsPushVisibilityTests exercises are ported in this PR, before the file is deleted, into the shaper's own tests (for example SyRF.ProjectManagement.Core.Tests/ProjectStatistics/Visibility/MembershipScreeningVisibilityShaperTests.cs), so the surviving HTTP callers keep their coverage. R0.1 ships releases before R4, so the port cannot wait for R4. These cases encode the legacy shaper's rules, which disclose more than the materialised read does (the materialised filter withholds peer rows without ViewMemberships, where the legacy shaper names or anonymises them; #3642). R4 must not inherit them as its expected output; R4's expected output is the materialised HTTP read for the same caller (R4 AC 4.1.3).

E2E: e2e/authority/tests/authority-m1a-application-roles.spec.ts:697-799 (calls at 697, 734, 771, 781 and 799) invokes SubscribeToProjectFullStats on two replicas and asserts ProjectStatsNotification pushes. It is the cross-replica M4b proof for a member-content payload stream, and it fails after R0.1. R0.1 rewrites it, for example to prove the same cross-replica late-query barrier on the ProjectNotification stream only, with the application-authority owner's agreement recorded in the PR.

Docs updated in the same PR: docs/features/materialized-project-statistics/STATUS.md:417 ("old push") and :939-942 (coordinated with the statistics session, which owns STATUS); phase0-calculation-consumer-catalogue.md:102-103, 910, 935, 1081-1094 (including option "© Wire SubscribeToProjectFullStats up"); docs/how-to/application-authority-mode.md:232.

Server only; the harmless web listener (ProjectStatsNotification → stageActions.receivedFullStats) goes with the client plan.

#	Criterion	Verification
0.1.1	Invoking SubscribeToProjectFullStats → HubException "method does not exist"; nothing aggregates	Hub unit test
0.1.2	Every other hub method and message still works	Existing SignalR tests (C1), with the 9 test files above translated or deleted
0.1.3	Web build and specs unchanged	CI Test Web (Angular)
0.1.4	A grep of src (*.cs) for SubscribeToProjectFullStats, ProjectStatsNotification and RealtimeSurface.FullStatistics → no match	grep in the PR body
0.1.5	Every visibility case from FullStatsPushVisibilityTests → has a ported equivalent against MembershipScreeningVisibilityShaper before the file is deleted; the PR lists the mapping	Unit (MembershipScreeningVisibilityShaperTests) plus the mapping table in the PR body
0.1.6	The rewritten authority-m1a-application-roles cross-replica test → passes, and still proves a member-content stream is refused at delivery after revocation on the other replica; the application-authority owner has approved the rewrite	bash e2e/authority/run.sh (hermetic); approval in the PR
0.1.7	The listed docs → no longer describe the legacy push as live or as an option to wire up	Docs review; validate-docs --skip-indexes

PR-R0.2 Per-event logging (RT7; effort S): MongoRepositoryBase.cs:169-212 to Debug via [LoggerMessage].

#	Criterion	Verification
0.2.1	One change event with 10 subscribers → 0 Information-level log lines from the fan-out	Log-capture unit test

R1: the v2 pipeline behind realtimePipelineV2, serving today's v1 hub (dark; flag off)

Restructured 2026-10-05 (Chris decision 2). The readers (R1.1) and the registry and dispatcher (R1.2) first deliver today's v1 messages, with the same names and payloads, to subscribers registered through today's v1 hub methods. They replace the Rx pipeline (AggregateRootEntitySubscriptionManager) behind the server-only flag realtimePipelineV2, so production can leave Rx without waiting for hub v2. Hub contract v2 (R1.3b) is a later step, for contract changes only. PR IDs are kept as they were, because other plans cite them; §8 gives the new order.

PR-R1.1 Change feed readers (RT1, RT2, RT6, RT11; effort M). Files: - new src/services/api/SyRF.API.Endpoint/Realtime/ChangeFeed/* (reader, EntityChange, projections, health, metrics); - API Program.cs registration (and, with the flag on, no registration of the Rx change-streams check, §4.6); - env-mapping.yaml flag realtimePipelineV2.

#	Criterion	Verification
1.1.1	Flag on; a Study save → exactly one EntityChange with id, project id, version, and nothing outside the projection	Testcontainers replica set
1.1.2	Primary stepdown mid-stream → the reader resumes from its token, and no change committed in between is missed	Testcontainers (replSetStepDown)
1.1.3	Resume token rejected (code 286 or 260) → the stream restarts at now, and one resync for that collection is emitted (ResyncRequired for v2; a fresh v1 message per subscribed topic for v1, §4.7)	Unit (fake cursor)
1.1.4	A reader retrying (any number of failures) → realtime-change-feeds reports Degraded, /health/ready stays 200, and the degraded metric rises; the cursor then opens with no events → Healthy within one poll	Unit (FakeTimeProvider); TestServer /health/ready
1.1.5	Flag off → no reader starts and no cursor opens; the Rx change-streams check is still registered and tagged live, as today	Unit over the registration
1.1.6	Wire size per Study change ≤ 1 KB (PROPOSAL)	Testcontainers command monitor
1.1.7	Flag on → no Rx change stream is opened on the host and the Rx change-streams check is not registered; realtime-change-feeds carries neither ready nor live; change-feed-stalled is tagged live	Unit over the registration
1.1.8	A reader unable to open its cursor for ≥ StalledRestartAfter (15 min, PROPOSAL) and the database ping succeeding → change-feed-stalled fails and /health/live returns 503	Unit (FakeTimeProvider, fake ping) + TestServer /health/live
1.1.9	The same stall with the database ping failing (a database outage) → change-feed-stalled passes, /health/live stays 200: no restart	Unit (FakeTimeProvider, failing ping)
1.1.10	A stall below StalledRestartAfter, or a cursor that opens at any point → the check passes, and the stall timer resets to zero (the next stall needs a full StalledRestartAfter again)	Unit (FakeTimeProvider)

PR-R1.2 Registry, dispatcher and IRealtimeClients, serving v1 (RT4, RT5, RT8, RT10 partly; effort M–L). Files: - new Realtime/RealtimeRegistry.cs, RealtimeDispatcher.cs, IRealtimeClients.cs; - NotificationHub.cs: the v1 subscription methods and OnDisconnectedAsync route into the registry when realtimePipelineV2 is on (names, policies and return values unchanged); - v1 message mapping moved off the cursor thread into the dispatcher.

#	Criterion	Verification
1.2.1	Flag on; 50 writes to project P within 1 s → each v1 P subscriber receives 1–2 ProjectNotification, the last carrying the newest state	Unit (FakeTimeProvider)
1.2.2	One Project change with 100 local subscribers → exactly 1 uncached Project read (and 1 investigator read) on that pod, not one per subscriber	Unit (read counter)
1.2.3	Membership disabled → that v1 subscriber gets today's v1 revocation behaviour and nothing later for P; other members are unaffected	Unit + Testcontainers (the existing NotificationHubDisabledMembershipRevocationTests cases, parameterised on the flag)
1.2.4	Private project Q changes → no listing subscriber without visibility receives Q in ProjectSummariesNotification	Unit (visibility matrix from NotificationHub.cs:1260-1267)
1.2.5	One socket's send throws → every other recipient is still sent to, and the dispatcher keeps running	Unit (C2)
1.2.6	Every v1 message (ProjectNotification, ProjectSummariesNotification, DataExportJobNotification, StudyPresenceUpdated) → same name and same JSON payload with the flag on as off; under enforced mode the per-delivery authority for payload streams still applies	Payload snapshot per message across both flag values (C1b); NotificationHubCurrentAuthorityTests parameterised
1.2.7	Disconnect → every registry entry for the connection is removed, export jobs included	Unit
1.2.8	The facade → a message sent through IRealtimeClients reaches a v1 connection now, and a v2 connection once R1.3b registers its hub context	Unit over the facade with both fake hub contexts

PR-R1.3a Extract review-session commands (RT12; effort M; behaviour-preserving, not flagged). Move NotificationHub.cs:143-1040 into ReviewSessionCommands (an application-level service) and keep thin hub methods.

#	Criterion	Verification
1.3a.1	NotificationHubStudyPresenceTests (1,999 lines) and ReviewSessionConnectionTests pass unchanged against the thin hub	dotnet test --filter
1.3a.2	Hub file ≤ 700 lines (PROPOSAL)	wc -l in the PR

PR-R1.3b RealtimeHub and contract v2 (§5; effort M; a later step, behind realtimeHubV2, which requires realtimePipelineV2 and stjHubProtocol). Files: - new Realtime/RealtimeHub.cs and IRealtimeClient.cs; - MapHub("/notifications/v2", o => o.AllowStatefulReconnects = true), on the STJ protocol the migration has configured (prerequisite); - ProjectStatisticsSubscriptionFilter.cs (accept SubscribeProject, record the hub kind; statistics-session sign-off); - contract doc and snapshot test.

#	Criterion	Verification
1.3b.1	Negotiate and connect at /notifications/v2 with a query token → connected; /notifications still works on the same host	WebApplicationFactory integration
1.3b.2	Snapshot of method names, parameters and message JSON → matches the committed contract; any change fails the test	Contract test
1.3b.3	A non-member calls SubscribeStudyPresence → HubException, and no registry entry is created	Hub unit test
1.3b.4	Unsubscribe calls for unknown topics → no exception	Hub unit test
1.3b.5	SubscribeProject → the ack carries contractVersion 2 and the project's current Audit.Version	Integration
1.3b.6	A v2 connection on an allowlisted pilot project receives ProjectStatisticsChanged	Integration (FEAT-024 harness)
1.3b.7	A v2 connection negotiates the json protocol → payloads are serialised by STJ with the migrated MVC options (camelCase; same enum and null handling)	Integration, comparing a hub message with the same DTO from an MVC response
1.3b.8	A client with stateful reconnect drops its socket for 5 s on the same pod → buffered messages arrive once, in order, with no new connection ID	Integration (HubConnection with WithStatefulReconnect)

PR-R1.4 Statistics invalidations via change feed (new 2026-10-05; effort M; behind realtimePipelineV2). Timing: after the gate (b) acceptance rerun and the 24-hour soak finish (about two days on Bramble from 2026-10-05). Neither measures notification delivery. Any write-path change after them needs a fresh 1-hour soak smoke. Review: the statistics session reviews the statistics parts of this plan and is a required reviewer on the R1.4 implementation PR; the statistics-side decisions in item 6 are theirs. Approved (2026-10-05): the statistics owner approved the statistics parts of R1.4 with the changes recorded in items 3, 4 and 6 and criteria 1.4.4, 1.4.5, 1.4.8 and 1.4.9 (statistics owner session, cross-session review 2026-10-05; see PR #4074). The flag-exclusive delivery in item 5 was added afterwards, from the PR review of c98ef818c, and needs the same owner's confirmation in the R1.4 PR.

Design (from the statistics session's review, 2026-10-05; code re-read in this PR's worktree): 1. Trigger: the slot collections, not the control row. One reader on pmProjectStatisticsNotificationOutbox (one coalescing slot per project + kind + family) and one on pmProjectStatisticsGlobalNotificationOutbox (mode and kill-switch transitions). Both watch insert and replace: slot writes are inserts and version-guarded ReplaceOneAsync (ProjectStatisticsRepositoryBase.cs:83-98, bulk :153, VERIFIED), so change events carry the full document and no updateDescription. The projection keeps _id, ProjectId, NotificationKind, MetricFamily and TargetClientInvalidationRevision (global: TargetGlobalClientInvalidationRevision, ProjectStatisticsGlobalNotificationOutbox.cs:34). The reader keeps the last target sent per slot in memory and sends only when the target rises. The web drops repeats by kind, family and revision (project-statistics-invalidation.service.ts:69-82), so a repeat after a restart is harmless. 2. Same message, same shape. ProjectStatisticsChanged(NotificationKind, ProjectId, MetricFamily, revision, …) exactly as ProjectStatisticsOutboxDispatcher.InvalidationFor and ProjectStatisticsNotificationSink build it today. The two provenance fields are filled from one control read per project event, as the sink does (ProjectStatisticsNotificationSink.cs:23-32, VERIFIED). Family and kind are preserved. 3. Same delivery checks. Per subscriber, from the existing FEAT-024 ProjectStatisticsSubscriptions: the SignalRConsumerEnabled flag and allowlist, then in Off mode the legacy uncached project and investigator View recheck, and in enforced mode the per-message IRealtimeDeliveryAuthority.DecideAsync with RealtimeSurface.StatisticsInvalidation. Denied → the subscription is removed. Every send is isolated (ProjectStatisticsChangedConsumer.cs:31-53,76-104, VERIFIED). A global slot event goes to every statistics subscriber. - Per-pod fan-out, as today. Each API pod runs its own two slot readers and delivers only to its own connections, from its own in-memory ProjectStatisticsSubscriptions. That is the same fan-out as today's per-pod temporary queues (ProjectStatisticsChangedConsumerDefinition: one temporary endpoint per pod), without the bus. 4. Catch-up instead of a resync message (revised with the statistics owner, 2026-10-05). Slots are durable, coalesced state, so the reader recovers from them, but differently for the two cases: - Fresh process start: send nothing. The reader opens the change stream first, with startAtOperationTime = the current cluster time, then seeds its last-sent map from one slot read, without sending. A deploy therefore does not make every client refetch. A browser that reconnects to the new pod already refetches on subscribe. The seed read must not hide a rise that happens between opening the stream and the read: it reads at the stream's start time (a snapshot read with atClusterTime), so any later rise arrives as an event above the seeded target. If the snapshot read is refused (for example SnapshotTooOld), the reader falls back to an ordinary read and then sends any post-open event whose target is ≥ the seeded value, so a rise in the gap is never dropped. A repeat at the same revision is harmless, because the web deduplicates. - Environment facts. The hermetic e2e Mongo runs as a single-node replica set (e2e/docker-compose.yml:35-37: --replSet rs0, with an rs.initiate health check), so change streams work there (VERIFIED). Atlas clusters (Preview, Cluster0) are replica sets by construction. Snapshot reads with atClusterTime outside a transaction need MongoDB 5.0 or later. Cluster0 runs 8.x according to the statistics owner (owner fact, not verified here). - Lost resume token inside a running process. The reader reopens at now, reads the current slots (for each project with a local statistics subscriber, bounded by ProjectStatisticsCatalogue.MaximumNotificationSlotsPerProject, plus the global slot) and sends each slot above its last-sent target exactly once. 5. One delivery path per host, chosen by realtimePipelineV2 (revised 2026-10-05 after the PR review; nothing is deleted in R1.4): - Flag off (the default and the rollback target): today's path keeps delivering, unchanged: ProjectStatisticsOutboxDispatcher → ProjectStatisticsNotificationSink → bus → ProjectStatisticsChangedConsumer on the per-pod temporary queue api-project-statistics-invalidations. The slot readers are not started. - Flag on: the slot readers deliver. On that host, ProjectStatisticsDispatchService skips its pass, in the same way it already skips when SignalRConsumerEnabled is off (ProjectStatisticsDispatchService.cs:25, VERIFIED), and ProjectStatisticsChangedConsumer acknowledges without sending. Consumer registration is unchanged (assembly scan, MassTransitHelpers.cs:45, VERIFIED), so the bus topology does not change before R3. - Why this is exactly one delivery per connection, even in a mixed fleet during a rollout or rollback. Both dispatchers are hosted only by the API: ProjectStatisticsNotificationRegistration.cs:13-15 and ActivityClaimRevocationDispatchService.cs:48-49 add the hosted services, and the only callers of DispatchPendingAsync are those two services (grep, VERIFIED). PM registers the dispatcher class in Lamar (ProjectStatisticsProductionRegistry.cs:262) but never runs it. So no PM-side dispatcher has to learn the API flag: each API host decides from its own deployed value. A flag-off pod's connections are served only by the bus path (its own consumer; a flag-off dispatcher somewhere in the fleet publishes). A flag-on pod's connections are served only by its own reader (its consumer is a no-op). - Rollback hazards (flag on → off), and the answer to each: - While the flag is on, nothing advances DeliveredThroughClientInvalidationRevision, so slots stay IsPending. After a rollback the dispatcher sends one invalidation per pending slot, at its current target. That is a one-off refetch per family, which clients deduplicate by revision. Accepted. - The backlog-ceiling fallback would then see PendingSinceUtc from the switch and mark every written family Stale with RetryExhausted. So its write gating is disabled in R1.4 under both flag values, leaving only the metric. The statistics owner already chose no write gating on notification lag (item 6), but this timing (R1.4, not R3) needs their confirmation. - The watermark bound is dropped in R1.4 under both flag values (item 6), so pending slots never pin reclamation. - Deleted in R3 (PR-R3.3), after production has run on realtimePipelineV2 for a Chris-approved period, as with the Rx deletion: the dispatcher and ProjectStatisticsDispatchService, the sink, the consumer and its definition, the api-project-statistics-invalidations temporary queues, the backlog-ceiling fallback, and the slot's lease, attempt and receipt fields. - Kept for good: every slot write (CoalesceTo), in the same transaction as the clock bump, at every writer site, so the gate (b) evidence stays valid. The sites the statistics session checked: ProjectStatisticsTransactionCoordinator.cs:1186-1200, ProjectStatisticsFoldWrites.cs:96-108, ProjectStatisticsRebuildService.cs:1370-1388 (the owner cited ~935, which is the #3826 retry note above the publication transaction; the slot write itself is at 1370-1388), ProjectStatisticsScopeFenceWriter.cs:228-250 and ProjectStatisticsWriteEpochLifecycleService.cs:578-620 (all VERIFIED). Further writers go through ScopeFenceWriter.CoalesceSlotAsync (SourceVisibilityFenceService:521, StagedOperationFenceService:114,215, DefinitionRewriteFence:178,226, DriftCheckService:686) or write directly (ModeEpochBatchService:268-278). A collection trigger covers them all without naming them. 6. Statistics-side decisions (theirs; recorded in the R1.4 PR): - Compaction watermark: DROP the IsPending bound (statistics owner session, cross-session review 2026-10-05; see PR #4074). It ships in R1.4 regardless of the flag: the owner judged the removal safe under either delivery path. It is still subject to the read-path proof below. The bound is dropped in both ProjectStatisticsDeltaCompactionService.cs:100-107 and ProjectStatisticsDeltaMaintenanceService.cs:178-182. Both hold their watermark at DeltaReplayFloorRevision while any slot IsPending. Once the dispatcher is gone, nothing advances DeliveredThroughClientInvalidationRevision, so IsPending would stay true after a slot's first write (ProjectStatisticsNotificationOutbox.cs:102-118) and pin both watermarks for good. The maintenance-service bound was found in this review; it was not in the owner's first list. - Proof required in the R1.4 PR: a grep and a test showing that no HTTP read path (current, bundle, history, receipts) replays the delta ledger on behalf of a client's invalidation revision. If none does, nothing a client can be told to refetch depends on the reclaimed ledger. - If one does, replace the bound in both services instead of dropping it: hold reclamation for Limits.MaximumRedeliveryWindow (24 h, ProjectStatisticsLifecycleLimits.cs:80, VERIFIED) after the slot's last target rise. The slot has no "last target rise" timestamp today (fields at ProjectStatisticsNotificationOutbox.cs:41-92, VERIFIED), so that route adds one, set by CoalesceTo in the same replace. That keeps the command count; the gate (b) counts (1.4.6) still have to be re-run. - ProjectStatisticsCatalogue.cs:7 cites the outbox for the slot-count bound. The bound and the NotificationScanCapacity check (DeltaMaintenanceService:178-181) stay valid, because the number of slots does not change; the comment is updated. - Backlog-ceiling fallback: RETIRED (statistics owner session, cross-session review 2026-10-05; see PR #4074). Today a family whose slot backlog age or attempt count crosses a limit is marked Stale with RetryExhausted, which stops its materialised writes (ProjectStatisticsOutboxDispatcher.cs:192-245, VERIFIED). Nothing replaces it: there is no write gating on notification lag. The alert is on realtime.change_feed.lag_seconds instead. Its write gating is switched off in R1.4 under both flag values (item 5, rollback hazard); its code is deleted with the dispatcher in R3.3. - Runbook note: families the old dispatcher already left Stale with RetryExhausted recover through the normal backfill and repair; no special step is needed. The R1.4 PR records this in the statistics runbook (phase2c-staging-proof-runbook.md). - RetryExhausted is shared, not dispatcher-owned. Rebuild publication uses it (#3826, ProjectStatisticsRebuildService.cs:991), as do daily observations (ProjectStatisticsDailyObservationProducer.cs:115, DailyProjectStatisticsConsumer.cs:84) and the admin 409 that clients key on (ProjectStatisticsAdminController.cs:325-329); all VERIFIED. Nothing new may treat it as dispatcher-owned (1.4.9). - Side effect: the dispatcher's claim writes were one source of the publication transaction's write conflicts (#3826, ProjectStatisticsRebuildService.cs:931-940). Once no host dispatches (every host flag on, and then R3.3), that source goes away; the retry stays, because source writes still coalesce the same slot.

#	Criterion	Verification
1.4.1	A family write for project P → each allowed P subscriber receives exactly one ProjectStatisticsChanged with the slot's NotificationKind, MetricFamily and target revision; a ProjectVisibility slot → kind ProjectVisibility, no family	Testcontainers (FEAT-024 harness)
1.4.2	A global slot advance (mode or kill-switch transition) → one GlobalInvalidation to every statistics subscriber on every pod	Testcontainers
1.4.3	Off, shadow and enforced modes → the same subscribers are removed or skipped as today; enforced mode calls IRealtimeDeliveryAuthority once per subscriber per message	Unit, ported from the ProjectStatisticsChangedConsumer tests
1.4.4	A slot replace that does not raise the target → no send. A fresh process start → nothing is sent for slots that existed before the stream opened; a change after the open → sent exactly once (including one landing between the open and the seed read). A lost token in a running process → each slot above its last-sent target is sent exactly once	Unit (fake cursor) + Testcontainers
1.4.5	The IsPending bound → dropped in both ProjectStatisticsDeltaCompactionService and ProjectStatisticsDeltaMaintenanceService (or replaced in both by the MaximumRedeliveryWindow grace, if the read-path proof finds a replay). This applies at R1.4 under both flag values (the ceiling's write gating is disabled at the same time; the two are coupled, statistics owner 2026-10-05). Reclamation advances with no dispatcher running, and under both flag values while slots stay pending, and a read immediately after the notified revision is still correct after compaction runs. The PR includes the grep and test proving no HTTP read path (current, bundle, history, receipts) replays the ledger for a client's invalidation revision	Unit + Testcontainers + grep; statistics-session approval
1.4.6	The gate (b) command counts → unchanged, because the slot writes stay in the source transaction	The gate (b) command-shape tests, unchanged
1.4.7	Flag off → statistics invalidations are still delivered through the old path (dispatcher → sink → bus → consumer), unchanged, and no slot reader starts. Flag on → the host's dispatch pass is skipped and its consumer sends nothing. In a two-host fleet with one host on and one off, each connection receives exactly one ProjectStatisticsChanged per target rise	Testcontainers (two API hosts, RabbitMQ) + the existing consumer and dispatcher tests, run with the flag off
1.4.7a	Flag on for a period, then rolled back to off → the dispatcher sends one invalidation per pending slot, and no family is marked Stale with RetryExhausted by the backlog ceiling	Testcontainers (FakeTimeProvider across the switch)
1.4.7b	Flag off → dispatcher backlog age raises an alert at the former backlog-ceiling thresholds (alert only, never a write gate). The rollback runbook notes that turning the flag off delivers at most one message per slot written since the switch, bounded by the catalogue's per-project slot limit, and clients just refetch	Unit (alert rule) + docs
1.4.8	Staging → a 1-hour soak smoke with the statistics write path under load passes, pages update after screening decisions, and the FEAT-024 staging preflight (six-hourly per-family parity) stays green for 24 h after the switch	Staging rollout check (statistics-session runbook)
1.4.9	The backlog-ceiling fallback's write gating → off under both flag values, and no write path is gated on notification lag; an alert on realtime.change_feed.lag_seconds exists. A grep → no new code treats RetryExhausted as dispatcher-owned; the runbook says how existing RetryExhausted Stale families recover	grep + alert rule review + runbook diff

PR-R1.5 Claim revocation and flag revision via change feed (new 2026-10-05; effort S–M; behind realtimePipelineV2; PROPOSAL for claim revocation, pending the review-eligibility owner, whose stream is paused). The same one-path-per-host rule as R1.4 item 5 applies; nothing is deleted before R3. - Claim revocation. - Flag off: unchanged. ActivityClaimRevocationDispatchService → ActivityClaimRevocationSink → bus → ActivityClaimRevokedConsumer on the per-pod temporary queue api-activity-claim-revocations. - Flag on: a reader on pmActivityClaimRevocationOutbox (insert) on that pod applies today's recheck (ActivityClaimRevokedConsumer.ShouldDeliverAsync, :37-70) and sends ActivityClaimRevoked to the owner's connections on that pod. On that host the dispatch service skips its pass and the consumer acknowledges without sending. The dispatcher is API-hosted only (R1.4 item 5), so each host decides from its own flag. - The intent write stays in the source transaction. - Rollback hazard: pending intents never expire (ActivityClaimRevocationOutbox.cs:42, VERIFIED), so while the flag is on they accumulate, and a rollback makes the dispatcher deliver the backlog. The recheck fails closed for claims re-acquired since (ActivityClaimRevokedConsumer.cs:72-80), and a revocation that is still true is harmless if late. Recommended (owner's call): with the flag on, the reader stamps DeleteAtUtc = now + Retention (7 days, ActivityClaimRevocationIntent.cs:28) with a conditional update, so the backlog stays bounded. - Flag revision. - Flag off: unchanged. The controller broadcasts locally (RuntimeFeatureFlagsController.cs:153), and other pods pick up the change on the 30 s refresh. - Flag on: a reader on runtimeFeatureFlagOverrides on every pod applies the snapshot locally and sends RuntimeFeatureFlagsRevision to local connections, and the controller skips its local broadcast. A mixed fleet can send one connection both a local broadcast (from a flag-off handling pod) and a reader send. The message is an idempotent revision number, so this is harmless. - Deleted in R3 (PR-R3.3): ActivityClaimRevocationDispatchService, ActivityClaimRevocationSink, the consumer, the api-activity-claim-revocations temporary queues, the controller's local broadcast, and (owner's decision) the intent's lease and attempt fields.

#	Criterion	Verification
1.5.1	Flag on; a committed claim revocation → the owner's connections on every pod receive one ActivityClaimRevoked; an owner who fails the recheck receives nothing	Testcontainers, two hosts
1.5.2	Flag on; an admin flag PUT on pod A → connections on pod B receive RuntimeFeatureFlagsRevision within 2 s (PROPOSAL), without waiting for the 30 s refresh	Testcontainers, two hosts
1.5.3	Flag off → claim revocations are still delivered through the old path and the controller still broadcasts, unchanged. Flag on → that host's claim dispatch pass is skipped and its consumer sends nothing; with one host on and one off, the owner's connections receive exactly one ActivityClaimRevoked	Testcontainers (two API hosts, RabbitMQ); existing claim-revocation tests run with the flag off

R2: parity proof and staging for realtimePipelineV2 (v1 hub)

PR-R2.1 Hermetic E2E parity (effort M). The e2e/ stack runs the realtime specs twice, with realtimePipelineV2 off and on (two-context project edit, listing update, export completion, presence, reconnect). The browser is unchanged: same endpoint and v1 messages.

#	Criterion	Verification
2.1.1	Both flag values → the same specs pass	E2E (hermetic)
2.1.2	Context B offline 10 s (PROPOSAL) while A edits P, then online → B shows A's edit without a page reload	E2E
2.1.3	API process restarted mid-session → the client reconnects, re-subscribes and converges	E2E (restart the API process in the stack)

R2.1 status (2026-10-14): the lane and specs are in PR #4143: bash e2e/run-local.sh --realtime --realtime-pipeline-v2 off|on (see e2e/README.md). The 10 s offline window in 2.1.2 is used exactly as proposed. R2.1 is done when both runs pass, as recorded in that PR. The lane is local only; CI's @full runs the parallel-safe specs with the flag off.

PR-R2.2 Staging switch (cluster-gitops staging/api/values.yaml flag realtimePipelineV2; effort S). Staging is not mission critical, so no window.

#	Criterion	Verification
2.2.1	7 days with realtimePipelineV2 on (PROPOSAL) → zero reader terminations; lag_seconds p95 ≤ 2 s; no API restarts attributed to realtime	Staging rollout check (GMP metrics)
2.2.2	Flag flipped back → the API restarts on Rx with no errors; clients keep working without a reload	Staging rollout check

Production realtimePipelineV2 = a separate Chris-approved step (cluster-gitops production values), after R2.2.

Hub v2 proof (later; after R1.3b and the STJ migration's PR-2.2). A hermetic E2E run and a staging soak with realtimeHubV2 on and the new client core pointing at /notifications/v2, reusing 2.1.1–2.2.2 against v2. Production hub v2 is another separate Chris-approved step.

R3: retire Rx, then v1

PR-R3.1 Delete Rx (effort L; moved earlier 2026-10-05). After realtimePipelineV2 has been proven in production for a Chris-approved period (D11). It does not wait for hub v2. Delete: - AggregateRootEntitySubscriptionManager.cs, ConnectionKeyedSubDictionaryCache.cs; - GetCachedCollectionChangeStream, ChangeStreamDictionaryByType, the Rx RetryWithExponentialBackoff in both libraries; - ICrudRepository.Get*NotificationStream (ICrudRepository.cs:36-37); - the Rx change-streams health check and ChangeStreamHealthTracker; - the System.Reactive package references; - the realtimePipelineV2 flag (the registry path becomes the only path).

The v1 hub methods stay; they already register into the registry. Translate or delete the Rx-based tests (CurrentAuthority, DisabledMembershipRevocation, StudyPresence, Logging, ApiSensitiveLogTests, and the Mongo.Common stream and retry tests).

#	Criterion	Verification
3.1.1	System.Reactive is gone from every csproj, and no file references System.Reactive	grep; build
3.1.2	The behaviours those tests asserted (revocation, delivery-time authority for payloads, presence) are asserted against the registry path	Test diff review
3.1.3	realtimePipelineV2 is removed from env-mapping.yaml, and the generator output is updated	Generator diff

PR-R3.2 Retire the v1 subscription methods (effort M). After hub v2 runs in production and MinUiVersion forces new bundles. The v1 hub is reduced to review-session delegation, or removed.

#	Criterion	Verification
3.2.1	No v1 subscription method remains; review-session commands still work on both hubs, or on v2 alone if v1 is removed	Hub tests

PR-R3.3 Delete the bus delivery paths (effort M; added 2026-10-05 after the PR review). Same gate as R3.1: after realtimePipelineV2 has run in production for a Chris-approved period. It can ship with R3.1. Delete: - statistics: ProjectStatisticsOutboxDispatcher, ProjectStatisticsDispatchService, ProjectStatisticsNotificationSink, ProjectStatisticsChangedConsumer and its definition, the api-project-statistics-invalidations temporary queues, the backlog-ceiling fallback, and the slot's lease, attempt and receipt fields (statistics-session review); - claim revocation: ActivityClaimRevocationDispatchService, ActivityClaimRevocationSink, ActivityClaimRevokedConsumer and its definition, the api-activity-claim-revocations temporary queues (review-eligibility owner); - the flag-revision controller's local broadcast.

#	Criterion	Verification
3.3.1	A grep → no ProjectStatisticsChangedConsumer, ProjectStatisticsNotificationSink, ProjectStatisticsDispatchService, ActivityClaimRevocationDispatchService, ActivityClaimRevokedConsumer or temporary-queue name remains; no bus publish of ProjectStatisticsChanged or ActivityClaimRevoked	grep in the PR body
3.3.2	The R1.4 and R1.5 reader criteria → still pass with the old path gone	Existing tests

R4: full-stats push on v2 (follow-up; staging only; sub-flag realtimeStatisticsPush)

Production rollout is not in scope of this plan (Chris, 2026-10-04: "leave prod for now"). R4 needs the FEAT-024 statistics session's agreement before it starts, because it reads their rows through a reader entry point they own and hooks their statistics delivery path. The statistics session reviews the R4 PR before it merges. R4 now follows R1.4 (2026-10-05): the bus consumer it used to hook is retired there, so the hook moves to the R1.4 slot reader.

The design was rewritten on 2026-10-05 to meet the ten correctness conditions in the statistics session's review (marked S1–S10 below). The earlier design (stored rows through ProjectStatisticsServingGate, shaped once per Named/Anonymised/None visibility class) is superseded: it would have pushed values older than the HTTP read in fold mode, and the three classes do not partition what the materialised read discloses.

PR-R4.1 ProjectStatisticsSnapshot push (effort M–L; part of the reader work is the statistics session's). Design: 1. Trigger, with no hot-path cost (S6). Revised 2026-10-05 for the one model: the R1.4 statistics slot reader, after its invalidation send, calls IRealtimeStatisticsPush.Enqueue(projectId, clientInvalidationRevision). Enqueue is O(1) and non-blocking (it updates an in-memory per-project entry and never awaits), because the reader is a single loop per pod and must never wait on a statistics read. Nothing is added to the save path. The push fires only after a materialised update has committed and its slot write has reached the change feed; there is no Study-save trigger. (The earlier text hooked ProjectStatisticsChangedConsumer, which does not deliver on flag-on hosts and is deleted in R3.3. R4 needs the flag on.) - Global changes (S5). A global slot event (ProjectId == null: mode or kill-switch transitions) never becomes a per-project read storm: the pod stops pushing and may send featureUnavailable to its statistics subscribers. Pushing resumes on the next per-project change whose fresh flags allow it. 2. Coalesce per project with a trailing window of 2 s (PROPOSAL). The pod skips the pass when the coalesced maximum trigger revision is ≤ the revision it last read for that project, because that earlier read already covered it (no data is lost: new pending entries always arrive with the next fold, at a higher revision). Coalescing keeps the maximum revision seen, because delivery can be out of order (S8). - Scope of the skip (#4050 review, 2026-10-05). The skip applies only to revision-triggered passes. It never suppresses the revoke path (a subscriber failing the item 7 check is dropped with SubscriptionRevoked, 4.1.6) or the omit path (a flag turned off, a bundle no longer Fresh, or a global event: the section is omitted or nothing is sent, 4.1.11). A skipped pass skips the read and the push only: the per-subscriber authority check and the per-pass flag check still run, whatever the revision. 3. Per-pod reads only with subscribers (S7). A pod reads only when it has at least one statistics:{projectId} subscriber, so N API pods make at most N reads per coalesced window. The registry is in memory per pod and is lost on restart, so the client resubscribes on reconnect (#3976, CR-4). 4. Flags per pass (S5, S9). Each pass resolves flags in a fresh scope, as the dispatcher does ("each pass gets fresh scoped flags"), so runtime overrides and the global kill switch apply at once. A section is included only when the same consumer flag the HTTP endpoint checks is on (for example materializedProjectStatisticsStageOverviewBundle, …ScreeningInfo, …ProjectOverview), together with …Pages, …Writes, …Serving, the family flags and the allowlist. Otherwise HTTP would serve legacy values while the push served materialised ones. 5. One read through the HTTP reader, fold overlay included (S1). The pass reads through the FEAT-024 bundle reader (ProjectStatisticsBundleReader, or IStageOverviewStatisticsQuery, which already serves the four Stage Overview families from materialised rows) in one pinned snapshot. In fold mode that serves stored row + pending entries, exactly as the HTTP read does (ReadScopesAsync overlay; async-point-fold-design.md §4). A stored-row read through ProjectStatisticsServingGate alone is not used. - Coherent-bundle rule: request exactly the selection sets the HTTP consumers request. If a bundle falls back, omit every section from it. Never mix a materialised section with a section HTTP serves authoritatively. - CORRECTED (statistics session review): "reviewer and annotation sections still come from the authoritative aggregation" is true only for FullStats (ProjectScreeningFullStatsSubstitution.cs:8-18). The Stage Overview bundle and ScreeningInfo consumers already serve materialised stageAnnotation and membership sections, and they are on in staging. R4 never calls the authoritative aggregation. - The read is bounded by the reader's caps (128 Studies / 1,024 pending entries, response budgets). 6. Per-caller authorisation and shaping through a new FEAT-024-owned reader entry point (S2). The materialised HTTP read is per caller: ReadCurrentAsync loads an authorization context for request.Caller inside the snapshot, applies ProjectStatisticsAuthorizationFilter.FilterCurrent and honours RequireActiveMembership (ProjectStatisticsBundleReader.cs:146-161). That filter withholds peer rows without ViewMemberships, where the legacy shaper names or anonymises them (#3642). Stage Overview adds the per-stage StageActivity.ViewAnnotationProgressGraph, and the project-wide read never carries annotation (StageOverviewStatisticsQuery.cs:217-226). So the three shaper classes are not a sufficient partition, and the per-class topic groups are dropped. Instead: - the statistics session adds a reader entry point (no existing method fits): one system-scoped snapshot read per pass, then the same FilterCurrent and shaper decision computed per subscriber, in memory, from that snapshot's Project; - a differential test proves the entry point's per-caller output is byte-equal to ReadCurrentAsync/the HTTP endpoint for the same caller and snapshot; - the payload has a stage dimension: a subscription is for a project and, optionally, a stage, because the Stage Overview read is per stage; - anonymised labels follow the HTTP contract (per response), so the client replaces a section wholesale. 7. Per-subscriber delivery authority in every mode (S3). After the read and before each send (the late-query barrier), each subscriber is checked for member-content scope (ProjectViewStudies/ActiveMembers) against the fresh Project facts, including the impersonation actor (RealtimeSubscriber.ImpersonationActorOf). The check applies in Off and shadow modes as well as enforced mode. The invalidation consumer's View-scope recheck (ProjectStatisticsChangedConsumer.cs:30-31,47-49) is too weak for a payload and is not reused. A failed check drops the subscription with SubscriptionRevoked. 8. Never push a non-Fresh or fallback value (S4). A section is omitted whenever its bundle is Fallback or Unavailable (disabled, fenced, inclusion or definition rewrite in progress, durable-mode disagreement, PendingOverlayUnavailable, Incompatible); the client keeps its HTTP read for it. Each section carries provenance: servedFrom: "materialized", the bundle watermarks/CheckpointId and PendingFingerprint, so a parity check can compare push and HTTP output. 9. No writes; clean callbacks (S6). The pass writes nothing: no receipts, outbox, control or cache rows. The snapshot transaction is closed before any SignalR send (invariant 14, "clean callbacks"). Metric tags stay bounded (C3). 10. Revisions and the equal-revision rule (S8). revision is the Int64 control ClientInvalidationRevision and globalRevision the Int64 GlobalClientInvalidationRevision, both read in the same pinned snapshot as the sections, not taken from the triggering message (the read can be up to 2 s later and include newer folds). They travel as strings and are always compared as integers, never as strings. The snapshot carries both because the client's pollCoherentStatistics tracks both (coherent-statistics-snapshot.ts:126-166). - Why the rule is safe (VERIFIED by the statistics session, 2026-10-05). A fold-path save only appends a pending entry; it moves no revision and publishes no ProjectStatisticsChanged. Only the fold worker removes entries, and it advances the clocks in the same transaction (ProjectStatisticsFoldWorker.cs:412, AdvanceClocks always increments ClientInvalidationRevision), then publishes. So at a fixed revision the CheckpointId is fixed and the pending set only grows, and new data always reaches clients at a strictly higher revision with a trigger. - Pod: per (project, stage, subscriber), drop a push whose revision is lower than the last one sent. At an equal revision, send only when the provenance (PendingFingerprint) differs; with the pass-skip in item 2 this almost never happens. - Which HTTP responses can be compared. Only the two coherent DTOs, StageOverviewStatistics and ScreeningOverviewStatistics, carry checkpoint, clientInvalidationRevision and globalClientInvalidationRevision. Neither carries a pending fingerprint, and the reviewer-progress (ProjectReviewerProgressMetadata: readSource only) and Project Overview screening-stats responses carry no revision at all. R4.1 therefore pushes only the sections those two coherent DTOs serve; the reviewer-progress and Project Overview stores keep invalidate-and-refetch unless a later, statistics-owned PR adds revisions to their responses. - Decision (recommended): a value read over HTTP has unknown provenance. An equal-revision push for an HTTP-held value is never applied and triggers one HTTP refetch. The alternative, adding pendingFingerprint additively to the two coherent DTOs, is a statistics-owned wire change (regenerated client, fold design §4 "the wire DTOs are unchanged") for a case the pass-skip already makes rare; the measured cost of the refetch is at most one GET per push per client, below today's invalidate→refetch rate, and a refetch publishes nothing, so it cannot feed back. Revisit only if staging shows equal-revision refetches are frequent. - Client (the rule the client plan implements, RS.1): apply a push only when its project revision is strictly greater than the one held and its global revision is not lower than the one held, whether the held values came from HTTP or from a push; a push with a lower global revision is dropped. At an equal project revision a push never overwrites the held value; if the held value came from HTTP (unknown provenance) or from a push with a different pendingFingerprint, the client makes one HTTP refetch. Pages keep accepting an equal revision from HTTP, and an HTTP response with a lower revision than held is dropped and the held value kept (today it throws "Statistics revision regressed" and the page goes unavailable). Accepted transient: an HTTP GET issued before a push can return at the same revision with a smaller pending set and overwrite the fresher push; the next fold (steady-state lag under 1 s) heals it at revision+1. - While subscribed, an invalidation does not refetch the sections the last push carried (the push for that revision follows); sections the push omitted still refetch on invalidation. - Server note for the client's RS.1.9a fallback (#4050 review, 2026-10-05). The server does not promise that a push follows every invalidation. A pass can be skipped (item 2), a section omitted (items 4 and 8), or a pod restarted between the invalidation and the push. So the client bounds its wait: if no push for that revision arrives within N s of the invalidation (PROPOSAL N = 5 s, above the 2 s coalescing window plus one read), it refetches over HTTP (client plan RS.1.9a). The server sends nothing extra for this. 11. Subscribe ack (S7). The SubscribeProjectStatistics ack carries a snapshot read fresh by that pod for that caller, through the same entry point and checks; never a cached last-sent value. 12. Staging only by construction (S9). The sub-flag defaults to false and the production values are asserted absent (4.1.8). The catalogue entry requires realtimeHubV2, materializedProjectStatisticsSignalR, materializedProjectStatisticsServing and the per-section consumer flags (§6).

#	Criterion	Verification
4.1.1	Flag on, allowlisted project; one materialised update → each subscriber receives exactly one ProjectStatisticsSnapshot whose revision equals the control ClientInvalidationRevision read in the same snapshot as its sections	Integration (FEAT-024 harness + v2 hub)
4.1.2	30 invalidations for project P within 2 s, delivered out of order → 1 snapshot read per pod and 1–2 pushes per subscriber, the last carrying the highest revision	Unit (FakeTimeProvider, read counter)
4.1.3	For each caller class (a member with ViewMemberships; a member without it; a member without the stage's annotation-graph permission; an impersonating administrator), project-wide and per stage → the entry point's output is byte-equal to ReadCurrentAsync/the HTTP endpoint for that caller on the same snapshot. The legacy shaper cases ported in R0.1 are not used as expected output	Differential unit + Testcontainers test (statistics-session-owned entry point)
4.1.4	An older revision arrives after a newer one → no push. A coalesced trigger revision ≤ the last revision read → no read, no push. Equal revision with the same pendingFingerprint → no push. Equal revision with a different fingerprint → a push is sent, and the client refetches over HTTP without overwriting its held value (also when the held value came from HTTP). Every push carries both revision and globalRevision from the same snapshot	Unit (pod) + client store specs (RS.1.2, RS.1.8)
4.1.5	A bundle in each Fallback/Unavailable state listed in item 8 → every section from that bundle is absent; no authoritative aggregation runs; no push mixes a materialised section with an authoritatively served one	Unit (aggregation spy) + Testcontainers
4.1.6	A member loses ProjectViewStudies, is disabled, or an impersonation actor lacks scope between the read and the send, in Off, shadow and enforced modes → no snapshot is sent and the subscriber receives SubscriptionRevoked; a permission grant → the next push matches the HTTP read for the new permissions	Unit
4.1.7	realtimeStatisticsPush off, or the project not allowlisted → SubscribeProjectStatistics returns a denial and nothing is read	Hub unit test
4.1.8	The production values file → realtimeStatisticsPush absent or false; production rollout is not in scope of this plan	Governance (cluster-gitops review)
4.1.9	(S10) Staging, 7 days with the sub-flag on (PROPOSAL), checked per caller class, including a member without ViewMemberships and a member without the stage annotation-graph permission → zero pushes with a revision lower than one already delivered, and the Stage Overview figures match the HTTP read for that caller after each push	Staging rollout check
4.1.10	Fold mode with pending entries → the pushed values equal the HTTP read including the overlay, and the provenance carries PendingFingerprint and CheckpointId	Testcontainers (FEAT-024 harness)
4.1.11	A consumer flag the HTTP endpoint checks (or Serving, Pages, a family flag or the allowlist) is turned off by runtime override → the next pass omits that section, or sends nothing, with no restart. A global ProjectStatisticsChanged (ProjectId == null) → no per-project reads and no further pushes	Unit
4.1.12	A pass → only read commands (no writes); Enqueue returns without awaiting; the snapshot session is closed before the first send; metric tags are bounded	Unit + Testcontainers command monitor + meter tag test (C3)
4.1.13	A pod with no subscriber for P → 0 reads for P. A subscribe after a change that was not pushed → the ack carries the fresh value	Unit
4.1.14	The flag catalogue → realtimeStatisticsPush requires realtimeHubV2, materializedProjectStatisticsSignalR, materializedProjectStatisticsServing and the per-section consumer flags	Flag dependency-catalogue validation test
4.1.15	A push never includes a reviewer-progress or Project Overview section (their HTTP responses carry no revision)	Unit (shape test)

Later, optional (R5), revised 2026-10-05: if persistence PR-10's domain-event outbox (pmDomainEventOutbox) is on, a semantic notification (inbox, attention) may be driven by a reader on that outbox collection instead of the entity collection, still through IRealtimeClients. It never goes through the bus (decision 1). Not scheduled.

8. Order, coordination and risks

flowchart LR
  R11[R1.1 readers] --> R12[R1.2 registry + dispatcher, serving v1] --> R21[R2.1 E2E, pipeline off/on] --> R22[R2.2 staging realtimePipelineV2] --> PROD{{Chris: production realtimePipelineV2}} --> R31[R3.1 delete Rx]
  GB{{gate b rerun + 24 h soak done}} --> R14[R1.4 statistics via change feed]
  R12 --> R14
  R12 --> R15[R1.5 claim revocation + flag revision]
  RE{{review-eligibility owner agrees}} --> R15
  R01[R0.1] & R02[R0.2] --> R13a
  N3932[#3932 merged] --> R13a[R1.3a extract] --> R13b[R1.3b v2 hub, later]
  STJ[STJ migration PR-2.1] --> R13b
  R12 --> R13b --> HV2[hub v2 E2E + staging] --> PRODV2{{Chris: production hub v2}} --> R32[R3.2 retire v1 methods]
  PROD --> R33[R3.3 delete bus delivery paths]
  R14 --> R33
  R15 --> R33
  R14 --> R41[R4.1 statistics push: staging only]
  HV2 --> R41
  F24{{FEAT-024 session agrees}} --> R41

• R0.1, R0.2 and R1.1 can run in parallel; they share no files except MongoContext.cs (R0.1 dead code, R1.1 none).
• R1.2 and R1.3a run in parallel.
• Critical path (revised 2026-10-05): R1.1 → R1.2 → R2.1 → R2.2 → (Chris: production realtimePipelineV2) → R3.1 (Rx) and R3.3 (bus delivery paths), after the same Chris-approved period. With the flag off, R1.4 and R1.5 change nothing, so they can merge before the production step. Backend PR-5 is no longer on it: its change-stream part is dropped, and its other probe work is independent. Neither the STJ migration nor hub v2 is on it.
• R1.4 needs R1.2, plus the gate (b) rerun and the 24-hour soak (about two days on Bramble from 2026-10-05). It can ship to staging before or after R2.2; it reaches production only with realtimePipelineV2.
• R1.5 needs R1.2; its claim-revocation half needs the review-eligibility owner.
• Hub v2: the STJ migration (PR-2.1) and R1.2 → R1.3b → hub v2 E2E and staging → production hub v2 → R3.2.
• R4.1 follows R1.4 (its trigger) and the hub v2 staging proof, plus the statistics session's agreement and their reader entry point; they review the R4 PR before merge. It is off the MVP path.

Stream / PR	Shared files	Sequencing
Backend plan PR-5 (#3974)	SyrfHealthCheckExtensions.cs, Quartz Program.cs, charts (no longer MongoContext.cs or ChangeStreamHealthTracker.cs)	Revised 2026-10-05. The Rx change-stream fix is dropped as stopgap work; change-feed health and recovery move to R1.1. PR-5 keeps its change-stream-independent probe work. Change-feed health never gates readiness (D15): R1.1 adds a Degraded-only check and the gated change-feed-stalled liveness check (1.1.7–1.1.10). The Rx change-streams check stays as it is until R3.1 deletes it
Backend plan PR-8/PR-9 (#3978)	Program.cs:381, NotificationHub.cs:112-131	Independent; R1.3a rebases on PR-9
Notifications stack: #3932 (adds InboxInvalidationWorker, InboxChanged, edits INotificationHubClient and signal-r.service.ts); #3938, #3941–#3945, #3965 touch Program.cs and NotificationsController only	NotificationHub.cs, Program.cs	R0.1 and R1.3a wait for #3932 to merge (and rebase on #2469, which also edits the hub). The inbox worker moves onto ChangeFeedReader in a follow-up owned with that stream
FEAT-024 (statistics session)	Statistics/* (consumer, sink, dispatch service, subscriptions, filter), ProjectStatisticsOutboxDispatcher, ProjectStatisticsDeltaCompactionService, ProjectStatisticsDeltaMaintenanceService, ProjectStatisticsCatalogue, the bundle reader and IStageOverviewStatisticsQuery (read only), and a new FEAT-024-owned per-caller reader entry point for R4	The ProjectStatisticsChanged contract is unchanged. R1.4 (2026-10-05): they review the statistics parts of this plan and are a required reviewer on the R1.4 implementation PR; it lands after the gate (b) rerun and the 24-hour soak, and any later write-path change needs a fresh 1-hour soak smoke. They decide the compaction watermark (both services) and the backlog-ceiling fallback (R1.4 item 6). R1.3b needs their review of the v2 method name. R0.1 removes the legacy full-stats push; they confirmed on 2026-10-05 that FEAT-024 does not depend on it, and R0.1 edits their STATUS.md lines only in coordination with them. R4 needs their agreement on: the O(1) enqueue hook in the R1.4 statistics reader; the new FEAT-024-owned per-caller reader entry point and its differential test; the coherent-bundle and omitted-section rules; provenance; the equal-revision rule; and the sub-flag dependencies (materializedProjectStatisticsSignalR, Serving, the per-section consumer flags). They review the R4 PR before it merges
Application-authority owner	e2e/authority/tests/authority-m1a-application-roles.spec.ts:697-799 (the cross-replica M4b proof on the full-stats stream)	R0.1 rewrites that test (for example onto ProjectNotification) only with their agreement, recorded in the PR (0.1.6)
STJ migration (system-text-json-migration-plan-2026-10.md, parallel plan)	API Program.cs:390-400 (hub protocol), MVC JSON options, converters	Prerequisite of R1.3b. It also moves the v1 hub to STJ (same protocol name). Its payload round-trip tests cover the hub DTOs listed in §5
Review-eligibility owner (paused)	ClaimRevocations/*, pmActivityClaimRevocationOutbox	R1.5's claim-revocation half is a PROPOSAL until they agree, including what happens to the intent's lease and attempt fields
Persistence plan (PR-10 outbox; #3985 version bumps)	none	version stays advisory until #3985; optional R5 depends on PR-10
ADR-021 (Wolverine, S1 Jan–May 2027 PROPOSAL)	the two per-pod temporary consumers	Revised 2026-10-05: R3.3 deletes both per-pod temporary consumers (statistics, claim revocation) once production has run on realtimePipelineV2. If that lands before S1, it shrinks S1 and leaves S0.12 nothing to pin. Until then, S0.12 still pins them, because flag-off hosts use them. IRealtimeClients is bus-agnostic
Authentication migration session	SubjectAdmissionHubFilter, query-token auth	D8 suspension abort needs their sign-off; /notifications/v2 reuses DirectBearerSchemeSelector unchanged
Client plan (parallel)	contract §5	The client core runs on the v1 hub, whose message names stay unchanged under realtimePipelineV2, so the client rewrite does not depend on the pipeline switch. It adopts v2 (client R3) after R1.3b

Risk	Mitigation
Missed events during the dual-path period or a pod restart	At-most-once is explicit; acks with versions, ResyncRequired, client re-declaration; E2E 2.1.2–2.1.3
Two pipelines double the change-stream load during transition	They never run on the same host: with realtimePipelineV2 on, no Rx stream opens (1.1.7); staging metrics in 2.2.1
A change-feed problem takes pods out of service, or restarts the whole fleet	Change-feed health is never part of readiness (Degraded + alert, 1.1.4). The only restart path is the gated change-feed-stalled liveness check, which requires a successful database ping (1.1.8–1.1.10)
A reader stalls forever while the database is fine	change-feed-stalled restarts that pod after StalledRestartAfter (1.1.8)
R1.4 stops delta reclamation, because IsPending never clears without the dispatcher	The bound is dropped in both services, with a read-path proof, or replaced by a 24 h grace (1.4.5)
Every deploy makes every statistics client refetch	A fresh start seeds the last-sent map without sending (item 4, 1.4.4)
version unreliable before #3985	Clients reload on any invalidation; patch-in-place only after #3985 (client plan rule)
Listing visibility filter leaks a private id	1.2.4 matrix test, built from the legacy predicate at NotificationHub.cs:1260-1267
The revocation stance for invalidations differs from M4b's per-delivery barrier	Payload messages keep the barrier; invalidations carry ids only; D8 records the auth-session sign-off
Stateful reconnect stalls sends when the buffer fills	Small messages; buffer size measured in the hub v2 staging proof; behaviour when full NOT VERIFIED, so R1.3b adds a test
Affinity cookie missing on a cross-origin negotiate, so connect lands on another pod	Already required by v1 and working in staging with 2 pods; R2.2 checks ingress logs (NOT VERIFIED for production)
Hub-file conflicts with in-flight PRs	R1.3a after #3932/#2469; gh pr list re-checked before each PR
The STJ prerequisite slips and delays R1.3b	R1.1, R1.2 and R1.3a proceed dark meanwhile; the client pilot runs on v1 behind RealtimeStore
The STJ switch changes v1 payloads (converter gaps, casing, enums, Int64)	Owned by the STJ plan's round-trip and parity tests; 1.3b.7 compares hub and MVC serialisation
R4 serves a statistics figure the HTTP read would not (stale overlay, wrong flags, fallback mixed in)	It reads through the same bundle reader as HTTP, overlay included, with the coherent-bundle rule and the same consumer flags per pass; non-Fresh bundles are omitted (4.1.5, 4.1.10, 4.1.11); staging check 4.1.9 per caller class
R4 leaks peer rows or annotation progress a caller cannot see over HTTP	Per-caller entry point with a byte-equality differential test (4.1.3); per-subscriber member-content check in every mode (4.1.6); the legacy shaper rules are not R4's expected output
A push overwrites a newer HTTP read at the same revision	Strict > on the client; at an equal revision a differing provenance triggers an HTTP refetch, never an overwrite (4.1.4)
Anonymised labels differ between pushes, so the client cannot diff rows	Same as the HTTP contract (labels are per response by design); the client replaces the section wholesale

9. Decisions needed from Chris

• D1. Paradigm. Replace Rx.NET with BackgroundService readers, a bounded Channel, a registry and a dispatcher, first behind today's v1 hub (realtimePipelineV2). Rx stays only until R3.1. Recommended.
• D2. Trigger: DECIDED (Chris, 2026-10-05). One realtime model: change-feed readers are the only trigger, including statistics invalidations (slot collections, R1.4). No bus path for SignalR delivery (§4.7). The earlier "hybrid" recommendation is superseded.
• D2a. Claim revocation onto a reader (R1.5). Recommended, pending the review-eligibility owner.
• D3. Message kinds. Invalidation for project details and listings; small data for presence, export status and claim revocation (§4.4). Recommended.
• D4. Backplane. None. Revisit if API pods exceed 10, if a producer must reach sockets without a per-pod source, or if per-pod change-stream cost shows in metrics. Recommended.
• D5. Retire SubscribeToProjectFullStats now, unflagged (no web caller since 2026-04-17). Stays as recommended: retire it. The statistics session confirmed on 2026-10-05 that FEAT-024 does not depend on it. The rewrite of the cross-replica E2E proof needs the application-authority owner's agreement (R0.1). Its replacement is R4, built on a different design.
• D6. Versioning. New typed hub at /notifications/v2, additive within v2, v3 path for breaks. Recommended.
• D7. Stateful reconnect on v2 (the client opts in through the client plan), treated as an optimisation only. Recommended.
• D8. Authorization for invalidations. Revocation-driven, with no per-delivery read. Abort a suspended subject's connections. Payloads keep the M4b barrier. Needs the authentication session's agreement. Recommended.
• D9. Contract sharing. Committed doc plus .NET snapshot plus a web spec over a JSON sample (recommended), versus TS code generation (TypedSignalR) as a later option.
• D10. PROPOSAL values:
• coalescing windows 1 s / 2 s / 1 s / 250 ms;
• StalledRestartAfter 15 min, and the degraded-time alert at 5 min (§4.6);
• ≤ 1 KB per Study change;
• hub ≤ 700 lines after extraction;
• E2E outage 10 s;
• staging soak 7 days with lag p95 ≤ 2 s.
• D11. Rx and bus-path deletion gate (revised 2026-10-05). How long production runs with realtimePipelineV2 on before R3.1 deletes Rx and R3.3 deletes the old statistics and claim-revocation delivery. Recommended: 4 weeks without a rollback. The v1 subscription methods go later (R3.2), after hub v2 is in production.
• D15. Change-feed health: DECIDED (Chris, 2026-10-05). Not part of readiness (Degraded plus a metric and alert). A separate gated liveness check, change-feed-stalled, restarts a pod only when a reader has been unable to open its cursor for ≥ StalledRestartAfter (15 min, PROPOSAL) and a database ping succeeds (§4.6). This replaces the earlier "readers report to readiness only" design.
• D14. realtimePipelineV2 before hub v2 (Chris decision 2, 2026-10-05). Production leaves Rx on the v1 hub first; hub v2 follows for contract changes only. Recorded; the flag name matches the implementation PR.
• D12. Hub protocol: DECIDED (Chris, 2026-10-04). System.Text.Json across the API, as a prerequisite planned separately; v2 uses AddJsonProtocol with the migrated MVC options.
• D13. Full-stats push: DECIDED (Chris, 2026-10-04). R4, staging only; production not planned. Still open: approve the realtimeStatisticsPush sub-flag, which requires realtimeHubV2, materializedProjectStatisticsSignalR, materializedProjectStatisticsServing and the per-section consumer flags (recommended over reusing realtimeHubV2; §6), and the PROPOSAL 2 s coalescing window. The R4 design was rewritten on 2026-10-05 to the statistics session's ten conditions (per-caller reads, fold overlay, equal-revision rule).

10. Not verified

• Production change-stream count, event rates and lag: no production metrics were read, and the API metrics PodMonitoring is staging-only until promotion.
• Whether Publish().RefCount() stays terminated after a non-retryable error (RT1): no longer worth proving, because the Rx path is replaced rather than fixed (2026-10-05).
• The flag name realtimePipelineV2 in the implementation PR: stated by the orchestrator; no local worktree or open-PR search shows it yet (2026-10-05).
• Whether the liveness probe's timeoutSeconds (1 s in the chart, values.yaml:283-289; production overrides not read) leaves room for the gated check's ping. R1.1 sets the ping timeout below it, and backend PR-5 proposes 3 s.
• The gate (b) rerun and soak timing (about two days on Bramble): stated by the statistics session, not observed here.
• Cluster0's server version (8.x per the statistics owner; not read here). R1.4's fresh-start seed read needs 5.0 or later for atClusterTime snapshot reads, and item 4 keeps the "send when target ≥ seeded" fallback for a refused snapshot.
• Stateful reconnect being WebSockets-only, and its behaviour when the buffer is full: secondary sources only. (Protocol support is now VERIFIED; see CR-6.)
• The exact signature of the new FEAT-024-owned per-caller reader entry point for R4 (no existing method fits; the statistics session owns it), and the exact ProjectStatisticsSnapshot field names.
• Production affinity on cross-origin negotiate: inferred from the ingress template, not observed.
• The Cluster0 MongoDB server version (it matters only if $changeStreamSplitLargeEvent were needed; the projections make it unnecessary).