Runtime feature releases: production policy, release records, docs and announcements¶

1. Status and boundary¶

This is the repository's plan for Chris's functional proposal of 15 October 2026 ("SyRF runtime feature releases functional implementation plan"). It turns that proposal into increments A–H, maps them onto the flag architecture roadmap P1–P10, and fixes interfaces, storage, tests and the decisions against the code on main.

Planning only, apart from the environment binding and the index migrations. The binding record, the bind mode and the RequiresBinding pre-write check (§6.3) are implemented in #4258, migrations 0009–0010 (the unique audit indexes, the first to require the binding, §6.4) in #4272, and migrations 0011–0020 (the indexes on the new runtime release collections, §6.4) in #4281. They run nowhere until an operator enables the pre-deploy Job and binds a database. Nothing else here is implemented, and nothing is deployed or authorised. No flag, staging state, production state or cluster-gitops value changes because of this document. Chris answered every decision in §16 on 15 and 16 October 2026. An answer settles the design; it does not authorise a production change, each of which still needs Chris's approval when it comes, and numerical targets stay trials until measured (OD-3, OD-20). The current rule still holds: GitOps values are the production source of truth and runtime overrides are non-production only (feature-flag rule). ADR-025 (Proposed) records the change of authority this plan asks for.

Production does not run this code yet. Production's API and PM run image 104eca9 (charts api-v9.44.0 and project-management-v11.43.0, cluster-gitops production/{api,project-management}/config.yaml). That build has no runtime-flag store, controller, shared reader or migration entry point (git grep for RuntimeFeatureFlag and MigrationEntryPoint at 104eca9 finds nothing). The ADR-023 pre-deploy migration Job is enabled in no environment (no migrations.enabled in cluster-gitops; pre-deploy plan M22). Every production statement below therefore starts after the continuous delivery plan's catch-up promotion has put production on main's runtime-flag code in Legacy mode with the migration Job enabled.

Deployment checks use a shared service, not a repository copy. On 15 October 2026 Chris refined decision D1 of the continuous delivery plan to an authenticated environment-state and feature-policy service, superseding option A (a maintained repository copy of production deployment inputs). The application, the guide, flag administration and deployment automation share one logical service boundary, built by extending the existing API and flag infrastructure, each through a scoped view. Staging and production expose the same contract and check workflow with separate state and credentials. §14 is the repository design for it, from Chris's revised proposal and its service addendum of the same day. Its numbers were accepted with §16 (OD-20, OD-22) for review after the first staging rehearsals; schema details are settled in the implementing PRs.

Evidence. The current-code matrix (Increment A) records what main does today, row by row, with file and line references at e5a3155c4. This plan cites it as "matrix P3.10" and so on.

Flag decision for this PR: none needed; documentation only.

2. MVP boundary, acceptance and critical path¶

MVP (Increments B and C): one selected, reversible, globally scoped control can be released, stopped and restored in production through the existing admin page, independently of a deployment, with the safety properties the architecture already requires. No targeting, no release records, no announcements.

MVP acceptance criteria (each is expanded into test cases in §15):

A production administrator previews and commits an On change for an eligible flag; every expected API and PM host acknowledges the new policy identity, and the admin page reports Active, without a deployment (RT-11, RT-13).
An emergency stop of the same flag reaches every expected host and new decisions use it within the accepted freshness objective, including while a docs or notification dependency is unavailable (RT-12, RT-40).
A host that has not acknowledged is reported unknown, never healthy; the page shows Degraded or Action required instead of Active (RT-11).
Concurrent edits conflict; a lost response is resolved by mutation ID; state and audit stay atomic (RT-01, RT-02).
Cold start, missed notification, revision regression, equal-revision corruption and restore never re-enable a stopped control (RT-03 to RT-06).
A binary rollback that could not honour the accepted policy is refused (RT-50).
After the catch-up promotion, production behaviour is unchanged until the deployment-controlled production policy mode is switched on by a Chris-approved cluster-gitops change (RT-10).
The same checker validates a candidate against a fresh staging report and then a fresh production report; a staging pass never stands in for the production check, and an incomplete report fails (RT-62, RT-63).

Shortest critical path: catch-up promotion (production on main's runtime-flag code in Legacy mode, ADR-023 Job enabled in staging (pre-deploy R1) and production) → environment binding record → authority envelope and lineage migration (P3) → mutation ID → host acknowledgement and expected inventory → baseline import for the eligible keys (P4) → compatibility report and scoped views → production policy mode, emergency stop, deployment constraint and production automation identity and production workload collector (C; OD-21, OD-23) → recovery exercise → production enablement of one control.

3. Assumptions this plan supersedes¶

#	Earlier assumption	Where	Replaced by
S1	Production ignores runtime overrides, so the browser value of a catalogued flag equals the rendered API value	CD plan §4.2 G2 step 4	Today production runs `104eca9`, which has no runtime-flag API at all, so the rule describes `main`'s code. After the catch-up promotion it is true only while production runs the legacy mode (`RuntimeFeatureFlags:ProductionPolicyMode=Legacy`, §5). After cutover the browser value is accepted baseline plus runtime policy; a legitimate runtime difference is not drift. The render comparison becomes conditional on the legacy mode (§14)
S2	The baseline refresh fails if the anonymous production snapshot disagrees with the render	CD plan §4.2, refresh step 4	Legacy mode only. After cutover: baseline-provenance and runtime-conformance checks (§14)
S3	After the catch-up promotion the live production snapshot becomes a second source for the baseline	CD plan §2	Legacy mode only. After cutover the live snapshot reflects runtime policy and cannot prove the baseline
S4	G8 expects staging to equal production's rendered effective flags plus listed overrides	CD plan §4.8	After cutover, staging is compared with a production-like synthetic profile plus listed pilots; production's live policy is never copied to staging (§14)
S5	G9 loads `production-effective-flags.json` into the E2E stack	CD plan §4.9	After cutover the profile is the accepted baseline plus synthetic policy fixtures, so an admin toggle never changes a test result
S6	Production stays read-only through P9; production policy is reviewed last (P10)	README §1 MVP criterion 6, §9 P10, §11 decision 1	Increment C moves the needed part of P10 ahead of P7–P9, for selected global controls only
S7	The admin page always shows production as read-only	README §7 step 1	Read-only until the production mode is on; afterwards production is editable for eligible controls by the capabilities in OD-2
S8	Freshness: 30 s reconciliation with a p95 under 2 s propagation objective	README §4 steady-state invariants	Trial objectives: decisions use authority checked within 30 s; enable confirmed within 60 s (§10). The 2 s p95 needs change streams and is deferred
S9	The first slice described in README §13 is the only shared-reader consumer	README §13	PM `StatisticsServing` mode also exists (matrix P5.1). README §13 now says so
S10	An informal idea of a "flag-aware What's New and user guide" in the delivery planning	CD plan conversation; no text on `main`	§12 of this plan. The CD plan now points here
S11	Release dates are the date a release note was written and shipped with the web image	`whats-new.json`, user-guide page titles	Release dates are confirmed production milestones (§8); the static feed stays for deployment-time content
S12	Production deployment inputs are compared from a vendored, maintained repository copy, refreshed by a bot PR (CD plan D1 option A, and the first draft of the proposal)	CD plan §3.2, §4.2 (vendored values, refresh workflow), §4.7–§4.8	Refined by Chris on 2026-10-15: trusted jobs fetch a versioned compatibility report from each environment's instance of the environment-state and feature-policy service (§14). Synthetic fixtures stay in the repository for untrusted PR tests
S13	A unique report or a fresh read of each source proves a consistent environment state	Implicit in the refresh workflow's "one resolved cluster-gitops SHA"	There is no atomic snapshot across GitOps, policy, workloads and domain state. A report carries a revision vector and a stability check; a changed or missing source makes it incomplete (§14.3)
S14	A preflight before deployment is enough	CD plan Phase 2 checklist	A policy change during a rollout can escape a one-off preflight. A deployment compatibility constraint, recorded by a trusted coordinator and enforced by the policy write path, closes that race (§14.5)

4. Ownership of state¶

Record	Authority	Notes
Flag catalog and consumer contract	Versioned source (`env-mapping.yaml`, the generated catalog, `consumer-manifest.json`)	Gains production eligibility, grain, safe state and stale behaviour (B)
Feature definitions	Versioned source, new `src/charts/syrf-common/feature-definitions.yaml`, generated to C# and TypeScript (D)	Stable feature and topic IDs; never derived from flag keys
Environment baseline	Accepted baseline identity inside the policy envelope after the P4 import	Staging and production may accept different baselines
Runtime policy	`runtimeFeatureFlagOverrides` in each environment's application database (existing), with the new envelope	Global settings, emergency stops; later targeting rules and cohorts
Environment compatibility report	Assembled on request by the environment-state and feature-policy service from the originating stores (§14)	A bounded, access-controlled run artifact; never a second authority and never a writable release-control surface
Deployment compatibility constraint	Recorded by the trusted deployment coordinator in the policy store (§14.5)	Read by every policy write; released by the coordinator (or an audited break-glass release), never lapses on its own (OD-22, §14.5)
Workload observations	Published by internal collectors and host heartbeats through a narrow scope (§14.2)	Observations, not authority; absent or stale means unknown
Release records, events, outbox	New collections in the same application database (§6)	API owns writes
Content bundle	Immutable reviewed content artifact (H; interim: the current Jekyll site and static feed)	Topic IDs, variants, publication class, assets
Discovery state	Per-user store (OD-14)	Separate from release dates

5. Increments A–H on the P1–P10 roadmap¶

Each increment keeps every existing safety acceptance criterion of the roadmap: mandatory revision checks, atomic audit, immutable snapshots, explicit baseline acceptance, recovery without silent re-enablement, domain readiness, operation boundaries and compatible rollback. Nothing below relaxes them; C adds production to the environments where they must hold.

Inc.	Roadmap phases	Scope on `main`	Exit evidence (numbered tests in §15)	Depends on
A Contract reconciliation	P0	This PR: current-code matrix, superseded assumptions, interfaces, feature and topic identities, the shared environment-report contract (§14.3), decisions	Matrix reviewed; ADR-025 Proposed	none
B Safe global runtime foundation	Rest of P1, P2, P3, P4; the audited subset of P5	Generated .NET catalog with eligibility, grain, safe state and stale behaviour (P1.1, P1.4); shared evaluator in `SyRF.Mongo.Common` used by API and PM (P2.3); environment binding record (§6.3); envelope with `environmentId`, `storeEpoch`, digests, explicit majority write/read concern and the reviewed legacy-lineage migration (P3.1–P3.2); mutation ID and receipt (P3.3); legacy API provider moved onto the shared snapshot or fail-closed staleness (P3.5–P3.6); browser seeding, reconnect fetch, decimal-string revision (P3.8–P3.9); per-pod local hint (P3.11); host acknowledgements and expected inventory (P3.10); baseline import for eligible keys (P4); provider mode renamed or aliased to the README's `Legacy\|Shadow\|Shared` (P3.7); the scoped service views, the compatibility report endpoint (schema v1, reporting unsupported fields as unknown) and the collector scope (§14)	RT-01 to RT-09 (with RT-04a), RT-11, RT-14, RT-60 to RT-65; two API and two PM replicas plus a browser in the hermetic stack; the same check contracts and checker pass against staging and against a production-like hermetic instance	A; for the staging exit, pre-deploy plan R1 (migration Job enabled in staging), because migrations `0009`–`0020` and `bind` run through that Job
C Limited production enablement	The needed part of P10; minimum P6 (P6.1 audit read, P6.2 preview digest, P6.5 split capabilities, P6.6 emergency stop)	Deployment-only `RuntimeFeatureFlags:ProductionPolicyMode=Legacy\|Runtime` (default `Legacy`) replacing the environment-name allow-list in the store, controller and shared reader (matrix P10.1–P10.3); production eligibility enforced per flag; emergency stop path; deployment compatibility constraint enforced by the policy write path (§14.5); production service identity for automation (OD-21); production workload collector (OD-23); recovery runbook and exercise; freshness trial (§10)	RT-10, RT-12, RT-13, RT-15, RT-40, RT-50, RT-66, RT-67; recovery exercise record; freshness measurements; production fetches its own fresh report	B; the CD plan's catch-up promotion (production on `main`'s runtime-flag code in `Legacy` mode); ADR-023 Job enabled in production; OD-21 (automation identity); OD-23 (production workload collector, a camarades-infrastructure change), without which no production report can be `Complete`
D Release records and guide contracts	New; uses P6 history	Feature definitions; release intent, record, events and outbox (§6–§7); availability endpoint; discovery preference; What's New projection of confirmed milestones; content manifest with publication classes; protected search and asset rules (contracts first, renderer in H)	RT-20 to RT-29, RT-41; dark deploy → release → stop → restore with a crash at every boundary	Contracts: A. Production path: C
E Project targeting	P7	Project-grain rules, domain prerequisites, project-aware availability and guide views	RT-30 to RT-32	C, D contracts
F Individual and combined targeting	P8	Trusted user and investigator mapping, combined project-and-user rules, consent lifecycle	RT-33 to RT-35	E
G Gradual rollout	P9	Stable basis-point cohorts, manual ramp and pause	RT-36, RT-37	E (F for personal grain)
H Reader migration and cleanup	Rest of P6, flag retirement	Design outputs, shared content compilation, public renderer, redirects, flag retirement	Link and content parity; accessibility; RT-28	D

Outside A–H, as the proposal says: P5a (statistics maintenance and use) keeps its own domain track and is never made safe by general production enablement; statistics maintenance is not a propagation demo. Cosmetic guide migration (H) does not block B or C.

flowchart LR
  A[A Reconciliation] --> B[B Global foundation]
  B --> C[C Limited production]
  A --> Dc[D contracts and prototype]
  C --> Dp[D production release path]
  Dc --> Dp
  C --> E[E Project targeting]
  Dc --> E
  E --> F[F Individual and combined]
  E --> G[G Gradual rollout]
  F --> G
  Dc --> H[H Reader migration]
  P5a[P5a statistics track] -.independent.-> C

6. Storage¶

6.1 Placement¶

Everything lives in each environment's application database, next to the existing runtimeFeatureFlagOverrides and runtimeFeatureFlagAudit collections, so policy, release intent and outbox can share one transaction and one backup. Nothing goes into the Identity database or a separate documentation database.

Environment	Database	Cluster	Configured by (evidence in §6.3)
Production	`syrftest` (production, despite the name)	Production Atlas cluster (`Cluster0`)	`production.values.yaml` sets the name; the cluster address is the chart default
Staging	`syrf_staging`	Preview cluster	`staging.values.yaml`
PR preview	`syrf_pr_{n}`	Preview cluster	`syrf-previews.yaml` ApplicationSet parameter
Local and E2E	Configured isolated database (`syrf_local`, per-stack E2E databases)	Local containers	Local configuration

6.2 Logical stores and indexes¶

Collection names follow the existing runtimeFeatureFlag* camelCase convention (no bounded-context prefix, because these are platform records, not domain aggregates). Field names are camelCase too, written as literal BSON keys as RuntimeFeatureFlagStore writes them. Migrations 0011–0020 fix the names of the collections and fields they index (§6.4); the discovery store's are still proposals.

Collection	Holds	Keys and indexes	Retention
`runtimeFeatureFlagOverrides` (existing)	Current policy envelope (`_id: "global"`, which also carries `constraintGuard`, excluded from `PolicyDigest`, §7.9) and the active deployment constraint (`_id: "deploymentConstraint"`, §14.5)	`_id`	Current state
`runtimeFeatureFlagAudit` (existing)	One entry per policy mutation	Add unique `{storeEpoch, revision}` and unique `{mutationId}`, both partial on documents carrying the new fields (legacy rows untouched)	Per OD-6
`runtimeEnvironmentBinding`	One binding document (`_id: "binding"`)	`_id`	Permanent; rotated only by an explicit rebind
`runtimePolicyHostAck`	Latest acknowledgement per host instance (fields in §10.2)	`_id` = instance UID; `{service, lastSeenAtUtc}` (`0011`)	No TTL: a silent host must stay visible as unknown until the inventory retires it
`runtimeFeatureRelease`	Current release record per release ID	`_id` = release ID; `{environmentId, status}` (`0012`). Unique `{environmentId, featureId, releaseKey}` is deferred to the release-record writer (increment D), which defines `releaseKey` (§6.4)	Permanent
`runtimeFeatureReleaseEvent`	Append-only milestone and lifecycle events	`_id` = event ID; unique `{mutationId, type}` (`0013`); `{releaseId, recordedAtUtc}` (`0014`); `{environmentId, featureId, recordedAtUtc}` (`0015`)	Permanent; never TTL
`runtimeFeatureReleaseOutbox`	Pending side effects	unique `{dedupKey}` (`0016`); `{state, nextAttemptAtUtc}` (`0017`)	Kept permanently, completed items included (Chris, 2026-10-17): they stay timestamped and queryable, and cost is negligible. Events never expire
Discovery store (OD-14)	Per-user seen and dismissed state	unique `{userId}` or `{userId, announcementId}`	Follows account deletion
`runtimeWorkloadObservation`	Latest workload facts from the collector (deployment generation, replicas, pod image digests, readiness)	`_id` = `{service}`; `{observedAtUtc}` (`0018`), where `observedAtUtc` is the collector's observation time	Current state only; staleness reported, never silently removed
`runtimeEnvironmentReport`	Retained compatibility reports needed to reproduce a check	`_id` = report ID; `{environmentId, collectionEndedAtUtc}` (`0019`); TTL `{deleteAfterUtc}` with `expireAfterSeconds: 0` (`0020`)	30 days (OD-20): the writer sets `deleteAfterUtc` to `collectionEndedAtUtc` plus 30 days (§7.8), so the period can change after the staging review without a migration. Access limited to the automation and Inspect scopes

Writes: the API owns every write. Readers use authorised API views; browsers never read MongoDB.

6.3 Verifying the environment binding¶

What the configuration says today (read-only; syrf e5a3155c4, cluster-gitops 5bdaa00d6 via the GitHub API, local cluster-gitops checkout untouched):

Environment	Database name	Cluster address	Environment name (`SYRF__RuntimeEnvironment`)	Source
Production	`syrftest`	`cluster0-pri.siwfo.mongodb.net` (chart default; production values do not override it)	`production`	cluster-gitops `syrf/environments/production/production.values.yaml:3-4, 19-20`; chart default `src/services/api/.chart/values.yaml:151-157` on `main` and the same at the deployed tags `api-v9.44.0` (`:90-96`) and `project-management-v11.43.0` (`:86-92`)
Staging	`syrf_staging`	`preview-pri.siwfo.mongodb.net`	`staging`	cluster-gitops `syrf/environments/staging/staging.values.yaml:4-5, 12-17`
PR preview	`syrf_pr_{prNumber}`	`preview-pri.siwfo.mongodb.net`; per-PR Atlas user secret `pr-user-mongodb-connection`	`preview`	cluster-gitops `argocd/applicationsets/syrf-previews.yaml:268-271`; `syrf/environments/preview/preview.values.yaml:24-25, 81-87`

SYRF__RuntimeEnvironment and SYRF__ActiveEnvironment are both computed from environment.name (env-mapping.yaml:65-68); the database name comes from mongoDb.databaseName (env-mapping.yaml:129-133). Value files are applied in the ApplicationSet order (argocd/applicationsets/syrf.yaml, value list) with ignoreMissingValueFiles: true.

Why the name is not enough. The chart default for every .NET service is database syrfdev on the production cluster address. If an environment's value file were missing or renamed, ArgoCD would render the chart default without error and the service would bind to syrfdev on Cluster0. Staging and preview share one cluster. A restored production snapshot in staging carries production's documents, including syrf_metadata.meta.environment: production, because that field is a best-effort SetOnInsert written by whichever pod starts first (DatabaseMetadataWriter.cs:239-246). The environment name defaults too: SYRF__RuntimeEnvironment and SYRF__ActiveEnvironment fall back to local (env-mapping.yaml:66, 68), and local is on both mutable allow-lists (RuntimeFeatureFlagStore.cs:471, SharedRuntimeFeatureFlagRegistration.cs:33). A missing production value file would therefore render a mutable local environment bound to syrfdev on Cluster0. None of these identify an environment on their own.

Design (B; built before any release-store migration). Implemented in #4258: the binding record, the bind mode, the chart values and the pre-write check in up (points 1–3 for up; ADR-023 §8 records the fingerprint format and exit codes). Migrations 0009–0010 (#4272) are the first to declare RequiresBinding, and the same PR makes the local E2E stack bind before up (§6.4). Previews do not run up yet (the migration Job refuses preview.waitForDatabase), so their database lifecycle Job binds when it switches to up. rebind (point 4) and the checks on release and policy writes (C) come later.

Binding record. runtimeEnvironmentBinding holds environmentId (production, staging, preview:pr-{n}, local:{stack}), databaseName, clusterFingerprint (the replica-set name and host suffix from the hello command, recorded at bind time), bindingEpoch (a new GUID), boundAtUtc, boundBy and an announcementDelivery mode: production only for the production binding, testSink for staging (an explicit test audience), disabled for previews, local stacks and restored copies (§12.5).
Explicit bind, never on start-up. A new entry-point mode SYRF_MIGRATION_MODE=bind writes the record once. It requires an expected environment, database and cluster, rendered from explicit chart values (migrations.binding.expectedEnvironment, .expectedDatabase, .expectedCluster) into the Job's own environment, with a helm unittest. They cannot come through env:, because the Job template drops every SYRF_MIGRATION_* key from .Values.env (_migrations-job.tpl, the hasPrefix "SYRF_MIGRATION_" filter). migrations.binding.bind: true renders the bind Job ahead of the up Job. It refuses if a different binding exists, and is run through a reviewed cluster-gitops change (production: Chris approves the specific PR). Local stacks and previews bind automatically, because they are created empty: the local E2E stack before every up (e2e/scripts/pm-migrations.sh, since 0009–0010), previews in their database lifecycle Job once it runs up.
Pre-write check. Before up applies any migration marked RequiresBinding, and before any release or policy write in C and later, the host checks all of: configured RuntimeEnvironment is the one the expected environment runs as (the part before the colon: preview:pr-{n} runs as preview, local:{stack} as local); the configured database name equals the expected name; the live hello fingerprint equals the expected cluster; and the binding record matches all three. The first two compare configuration with configuration; the binding record and the hello fingerprint are the checks that observe the real database. Any mismatch or a missing record stops with exit code 2 (usage) and writes nothing.
Rebind after restore. A restored copy carries the source binding, so the check fails until an operator runs rebind, which rotates bindingEpoch, records the source binding, sets announcementDelivery: disabled and starts a new policy lineage (README §3, "Rollback writes a new forward revision"). Release events and outbox deduplication keys include bindingEpoch, so replay from a restore cannot reannounce or reinstate anything.

The IsMutationAllowed name allow-list (RuntimeFeatureFlagStore.cs:467-487) and the shared reader's equivalent are replaced in C by this binding plus ProductionPolicyMode; environment names stop being an authority on their own.

6.4 Provisioning through ADR-023¶

Catalogue. Append to PmMigrations (the one catalogue for this database, already run by both the API and PM images). 0009 and 0010 are the audit indexes (#4272), and 0011–0020 index the runtime release collections (#4281). The next free version is 0021; recheck PmMigrations.cs and migrations.lock.json when implementing. A second catalogue in the same database is not supported today: the runner reads every syrf_migrations record and treats unknown IDs as applied by a newer binary (matrix E.2). Supporting one would need catalogue scoping in the state store, which is not worth it here.
Kind. Index migrations only (CreateIndexMigration). 0011–0020 index new collections, and createIndex creates the collection, so no new migration kind is needed. 0009–0010 index the existing runtimeFeatureFlagAudit collection. Both groups meet ADR-023's catch-up rollback rule as amended for OD-10 option C (decided 16 October 2026): an undo puts the database back exactly as it was before the step ran (an index the step created is dropped, an index it found already in place, recorded as adopted, is kept), so every migration after version 4 only has to be an index migration, on any collection. The runner change and the ADR-023 amendment land first, in #4255, before 0009. MigrationOnlyCollections no longer takes part in the rollback rule; it still governs start-up index creation. runtimeFeatureFlagAudit is listed there for that reason alone: its only writer builds no index, so the start-up governance tests cover it with PreDeploy off and on. The six collections 0011–0020 index are listed for the same reason: nothing writes them yet, their writers will build no index, and the governance test requires every index migration's collection to be either governed or migration-only. Reordering was never an option, because the catalogue is append-only (Catalogue_MatchesTheCommittedLockFile).
Forward rule. An Up runs while the previous release is still serving (ADR-023 rule 5), so a unique index must be one the previous release cannot violate. Each unique audit index is therefore partial on fields the previous release never writes: 0009 on { mutationId: { $exists: true } }, 0010 on { storeEpoch: { $exists: true } }. $exists rather than $type, so that a value of the wrong BSON type is still held to uniqueness instead of bypassing it. #4272 proves the rule in two API test classes:
RuntimeFeatureFlagAuditDocumentShapeTests: the documents BuildAudit and BuildGroupAudit write carry neither field.
RuntimeFeatureFlagAuditIndexForwardRuleTests, with both indexes applied on a replica set: legacy rows that share a revision insert without collision; the indexes build over existing legacy rows that repeat a revision; and today's RuntimeFeatureFlagStore keeps writing audit rows, even after its revision restarts at 1.

Each of those has a same-data positive control. Copies of the same legacy rows that share a mutationId, or a {storeEpoch, revision}, are rejected with duplicate key 11000. A non-partial unique revision index cannot be built over them.

0011–0020 index collections that no release up to this one reads or writes, so their indexes build over empty collections and nothing the previous release does can violate them. #4281 proves it in the PM data tests: - RuntimeReleaseCollectionsForwardRuleTests: no production C# source except PmMigrations.cs names any of the six collections, by literal or by catalogue constant. Its positive control is the same scan finding RuntimeFeatureFlagStore.cs, the audit collection's known writer. When a writer lands, this test fails on purpose and gives way to a document-shape test of that writer. - PmRuntimeReleaseIndexMigrationsTests, on a replica set: the unique event and outbox indexes accept a duplicate pair before the migrations and reject the same pair with 11000 after them, accept distinct keys, and reject a second document missing the keys.

The two unique indexes are therefore not partial: there are no legacy documents to leave out, and a document missing a key is indexed as null, so a second one fails closed instead of escaping the constraint.

A $jsonSchema validator would be a new migration kind and needs the rollback rule revisited first; the first slice validates in the API instead. - Sequence. Assigned in #4272, the first migrations to declare RequiresBinding: - 0009_runtimeFeatureFlagAudit_mutationId_unique: audit {mutationId}, unique partial, index mutationId_1_unique; - 0010_runtimeFeatureFlagAudit_storeEpoch_revision_unique: audit {storeEpoch, revision}, unique partial, index storeEpoch_1_revision_1_unique.

Assigned in #4281, one index per migration, each declaring RequiresBinding: - 0011_runtimePolicyHostAck_service_lastSeenAtUtc: {service, lastSeenAtUtc}, index service_1_lastSeenAtUtc_1; - 0012_runtimeFeatureRelease_environmentId_status: {environmentId, status}, index environmentId_1_status_1; - 0013_runtimeFeatureReleaseEvent_mutationId_type_unique: {mutationId, type}, unique, index mutationId_1_type_1_unique; - 0014_runtimeFeatureReleaseEvent_releaseId_recordedAtUtc: {releaseId, recordedAtUtc}, index releaseId_1_recordedAtUtc_1; - 0015_runtimeFeatureReleaseEvent_environmentId_featureId_recordedAtUtc: {environmentId, featureId, recordedAtUtc}, index environmentId_1_featureId_1_recordedAtUtc_1; - 0016_runtimeFeatureReleaseOutbox_dedupKey_unique: {dedupKey}, unique, index dedupKey_1_unique; - 0017_runtimeFeatureReleaseOutbox_state_nextAttemptAtUtc: {state, nextAttemptAtUtc}, index state_1_nextAttemptAtUtc_1; - 0018_runtimeWorkloadObservation_observedAtUtc: {observedAtUtc}, index observedAtUtc_1; - 0019_runtimeEnvironmentReport_environmentId_collectionEndedAtUtc: {environmentId, collectionEndedAtUtc}, index environmentId_1_collectionEndedAtUtc_1; - 0020_runtimeEnvironmentReport_deleteAfterUtc_ttl: {deleteAfterUtc}, TTL with expireAfterSeconds: 0, index deleteAfterUtc_1_ttl. This is OD-20's retention. The writer sets deleteAfterUtc to collectionEndedAtUtc plus 30 days (§7.8), so the period lives in the data and the review after the first staging rehearsals can change it without a migration. A changed expireAfterSeconds would need a collMod migration kind, which does not exist. A report without the field, or with a value that is not a date, is never removed (PmRuntimeReleaseIndexMigrationsTests).

Each is idempotent, adopts an identical existing index (which an undo then keeps), and drops on undo only an index its Up created. The tests show up without a binding stopping before 0011 with exit 2, building nothing and creating no collection, and then, once bound, applying exactly 0011–0020.

Not in 0011–0020: - The unique {environmentId, featureId, releaseKey} release index of §6.2. It is deferred to the release-record writer (increment D), with its forward-rule test. No record defines releaseKey (§7.3), so built now it would index every record as releaseKey: null, and a feature's second release in an environment (§8.4) would collide with 11000. Open item for D: define ReleaseKey on ReleaseIntent and ReleaseRecord (what makes a release "genuinely new", §8.4), then add the index. - No outbox TTL. Chris decided on 2026-10-17 to keep completed outbox items permanently, as a timestamped, queryable record of every release side effect (§6.2). No TTL index is planned. - The discovery store. Its shape is not settled (OD-14: unique {userId} or {userId, announcementId}, §7.6). It is a follow-up, indexed with its writer. - What migrations never do: seed release dates, import history, create the binding record, initialise the policy lineage, or write anything on pod start-up. The legacy-lineage migration (README §3) and the historical import (§8.5) are reviewed commands with dry-run diffs and receipts, run separately. - Readiness. Code that uses these collections checks the indexes exist, as reversible deletion does (MongoDeletionSchemaReadiness), and fails closed (release features unavailable) if they do not. The emergency stop is the exception: it must not depend on the new audit indexes (Q1, accepted 16 October 2026). - Where the Job runs. The pre-deploy Job is enabled in no environment today (pre-deploy plan M22), so none of these migrations, and no bind, can run anywhere until: staging enables it (pre-deploy plan R1); production enables it (inside the catch-up promotion PR, per that plan); previews switch their DatabaseLifecycle index-init Job to up (the template refuses migrations.enabled with preview.waitForDatabase, in _migrations-job.tpl). Until each happens, release features in that environment stay unavailable (OD-11). - Enabling the Job needs a binding first. Because 0009 requires the binding, enabling the Job in an environment must bind that environment first, either in the same cluster-gitops change or before it. - In the same change: set migrations.binding.expectedEnvironment, .expectedDatabase and .expectedCluster, with migrations.binding.bind: true, alongside migrations.enabled. The bind Job runs in sync wave −1, ahead of up. Copy the expected cluster from the live fingerprint that status prints. - Without a binding, up stops before 0009 with exit 2 and applies nothing, 0001–0008 included. The PreSync Job then fails, and the previous release keeps serving. - Local stacks bind themselves. e2e/scripts/pm-migrations.sh, run by run-local.sh and e2e-tests.yml, sets RuntimeEnvironment=local for three entry-point runs: 1. status reads the live fingerprint, which the binary itself computes; 2. bind binds to local:<E2E_STACK>, with the database from appsettings.e2etest.json (syrf_e2e); 3. up applies the pending migrations.

The binary still checks all four binding checks, and no new code path accepts an unverified cluster. e2e/scripts/pm-migrations.test.sh is its contract.

7. Interfaces against current code¶

Sketches, not final code. Existing types are named where they are extended. Revisions cross every HTTP boundary as decimal strings.

7.1 Policy identity (extends `RuntimeFeatureFlagSnapshot`)¶

// Global policy identity. Changes on every policy mutation, for any flag.
public sealed record PolicyIdentity(
    string EnvironmentId,      // from the binding record, e.g. "production", "preview:pr-4230"
    Guid BindingEpoch,
    Guid StoreEpoch,           // lineage; never compared for order
    long Revision,             // monotonic within StoreEpoch; JSON: "412"
    string CatalogDigest,
    string BaselineDigest,
    string PolicyDigest);

public sealed record RuntimeFeatureFlagMutation(   // existing record, extended compatibly
    bool? Value, long? ExpectedRevision, string Reason,
    bool IncludeDependencies = false,
    IReadOnlyDictionary<string, bool>? ConfirmedDependencies = null,
    Guid? MutationId = null,          // required once B ships; reuse with a different payload is rejected
    string? PreviewDigest = null,     // required in production and staging (C); optional only for local and E2E
    PolicyAction Action = PolicyAction.Set); // Set | Clear | EmergencyStop | ReleaseStop

7.2 Feature definition (generated from `feature-definitions.yaml`)¶

public sealed record FeatureDefinition(
    string FeatureId,                       // stable, e.g. "bulk-pdf-upload"; never a flag key
    string Owner,
    CapabilityExpression Capability,        // typed AllOf/AnyOf of FlagRequirement(key, value); no scripts
    FeatureGrain Grain,                     // Global | Project | Personal (allowed evaluation grains)
    IReadOnlyList<string> CompatibleWorkflowVersions,
    IReadOnlyList<string> TopicIds,         // permanent guide topic IDs
    FeatureLifecycle Lifecycle,             // Developing | Piloting | Released | Retired
    GuidanceRequirement Guidance);          // Required | Exempt(reason)  (OD-5)

Retiring a flag removes it from Capability through a reviewed change; the feature and its history stay.

7.3 Release intent, record and event¶

The release ID is per feature release and is never derived from the policy revision, so an unrelated flag change cannot create a release or an announcement.

public sealed record ReleaseIntent(
    Guid ReleaseId,                 // stable across pause, restore and expansion
    string EnvironmentId, Guid BindingEpoch,
    string FeatureId,
    ReleaseStage RequestedStage,    // LimitedRelease | GeneralRelease | PublicPreviewPublication
    AudienceSpec Audience,          // Global in C; Projects/Users/Cohort from E-G
    long ExpectedPolicyRevision, string PreviewDigest,
    string? ContentBundleDigest, AnnouncementRef? Announcement,
    DateTimeOffset? ScheduledForUtc,  // planned; displayed as Planned until confirmed
    ActorRef Actor, string Reason, Guid MutationId);
// RequestedAtUtc is assigned by the server clock, never by the client.

public sealed record ReleaseRecord(
    Guid ReleaseId, string EnvironmentId, string FeatureId,
    ReleaseStatus Status,           // Prepared | Applying | Active | Degraded | ActionRequired | Paused | Withdrawn
    IReadOnlyDictionary<ReleaseStage, Milestone> Milestones,
    long Version);                  // optimistic concurrency on the record

public sealed record Milestone(
    DateTimeOffset RequestedAtUtc,
    DateTimeOffset? ScheduledForUtc,
    DateTimeOffset? ConfirmedAtUtc,     // derived by the server when activation criteria hold
    DateTimeOffset? RecordedAtUtc,      // when the confirmation was written
    ConfirmationSemantics Semantics,    // ObservedActivation | ObservedAfterRecovery | Imported
    TimePrecision Precision,            // Instant | Date | Month | Unknown
    PolicyIdentity? ConfirmedPolicy);

public sealed record ReleaseEvent(
    Guid EventId, Guid MutationId, Guid ReleaseId, string FeatureId,
    string EnvironmentId, Guid BindingEpoch,
    ReleaseEventType Type,  // Prepared | PolicyCommitted | ActivationConfirmed | Paused | Restored |
                            // EmergencyStopped | PublicationRecorded | PublicationPending |
                            // AnnouncementQueued | AnnouncementCancelled | Corrected | HistoricalImported
    ReleaseStage? Stage,
    PolicyIdentity? ObservedPolicy,
    DateTimeOffset? OccurredAtUtc, TimePrecision OccurredPrecision,
    DateTimeOffset RecordedAtUtc,
    EventProvenance Provenance,     // Observed | Imported(source, reviewer) | Correction(ofEventId)
    EventResult Result, string? Detail);

7.4 Outbox item¶

public sealed record ReleaseOutboxItem(
    string DedupKey,  // "{environmentId}:{bindingEpoch}:{releaseId}:{stage}:{announcementId}:{announcementRevision}:{kind}"
    Guid SourceEventId, Guid ReleaseId,
    OutboxKind Kind,  // MaterialiseWhatsNew | NotifyInApp | PublishPublicContent | CancelAnnouncement
    OutboxState State,                  // Pending | Leased | Done | Cancelled | Failed
    int Attempts, DateTimeOffset NextAttemptAtUtc, LeaseInfo? Lease,
    PolicyIdentity GuardPolicy,         // rechecked against current policy before the side effect
    DateTimeOffset CreatedAtUtc, DateTimeOffset? CompletedAtUtc); // kept permanently: no expiry (§6.2)

7.5 Availability response¶

GET /api/features/availability?projectId={id} (authenticated; projectId optional; membership checked on the server). An anonymous variant returns the public subset for global features only. Responses are Cache-Control: private, no-store.

{
  "contextId": "sha256:…",
  "policy": { "environmentId": "production", "storeEpoch": "…", "revision": "412", "policyDigest": "…" },
  "freshness": { "checkedAtUtc": "2026-11-02T10:15:04Z", "state": "Current" },
  "features": [
    {
      "featureId": "bulk-pdf-upload",
      "availability": "available",
      "reason": null,
      "workflow": "v2",
      "contentRevision": "guide-2026-11-01",
      "activation": "Active"
    }
  ]
}

availability is available, unavailable or unknown. reason is a bounded code: notReleased, excluded, stopped, prerequisiteMissing, permission, policyUnavailable, staleAuthority. contextId hashes environment, user, project, membership version and policy identity; a browser discards a response whose contextId no longer matches its current context.

7.6 Discovery preference¶

PUT /api/account/announcements/{announcementId}/state with { "revision": 2, "projectId": null, "seen": true, "dismissed": true }; GET returns the caller's entries. Stored as { userId, announcementId, revision, projectId?, seenAtUtc?, dismissedAtUtc? }.

7.7 Administration endpoints¶

Route	Purpose	Capability (OD-2)
`POST /api/runtime-feature-flags/{key}/preview`	Side-effect-free plan: before/after, dependencies, induced pins, revealed baseline, expected hosts, `previewDigest`	Change policy
`PUT /api/runtime-feature-flags/{key}` (existing)	Commit; requires `previewDigest` and `mutationId` in production and staging (same workflow); optional only for local and E2E	Change policy
`POST /api/runtime-feature-flags/{key}/emergency-stop`	Forward revision forcing the catalog's safe state; no content or release checks	Emergency stop
`GET /api/runtime-feature-flags/mutations/{mutationId}`	Resolve an unknown commit outcome	Change policy
`GET /api/runtime-feature-flags/audit?cursor=`	Paginated audit history	Inspect
`GET /api/runtime-feature-flags/fleet`	Expected hosts, acknowledgements, freshness, state	Inspect
`POST /api/feature-releases/{featureId}/preview`	Release preview: policy plan plus guidance and host checks	Manage releases
`POST /api/feature-releases`	Commit a release intent	Manage releases
`GET /api/feature-releases/{releaseId}`	Record and events	Inspect
`POST /api/environment/compatibility-reports`	Assemble a compatibility report (§7.8); returns `202` with the report ID, then the report	Automation read scope
`GET /api/environment/compatibility-reports/{reportId}`	Fetch a retained report	Automation read scope or Inspect
`PUT /api/environment/deployment-constraints/{rolloutId}`	Record or extend the deployment compatibility constraint (§7.9). At most one is active per environment: a `PUT` for a different `rolloutId` while one is active returns 409	Deployment coordinator scope
`DELETE /api/environment/deployment-constraints/{rolloutId}`	Release the constraint after rollout completes or rolls back; refused if `rolloutId` is not the active one	Deployment coordinator scope
`POST /api/environment/deployment-constraints/{rolloutId}/break-glass-release`	Release a constraint whose coordinator has died; requires a reason, is audited in `runtimeFeatureFlagAudit`, and is reported on the fleet view	Administrator only (separate break-glass capability, OD-22)
`POST /api/environment/observations`	Collector publishes workload observations (§14.2)	Collector scope; cannot read or change policy

7.8 Compatibility report¶

public sealed record EnvironmentCompatibilityReport(
    int SchemaVersion,                    // 1; validators accept N and N-1 during an upgrade
    Guid ReportId,
    string EnvironmentId, Guid BindingEpoch,
    DateTimeOffset CollectionStartedAtUtc, DateTimeOffset CollectionEndedAtUtc,
    DateTimeOffset ExpiresAtUtc,          // proposal: 15 minutes after collection; stale for checks, not deleted
    DateTimeOffset DeleteAfterUtc,        // retention (OD-20): CollectionEndedAtUtc + 30 days; the TTL index of
                                          // migration 0020 removes the retained report after it (§6.2, §6.4)
    ReportCompleteness Completeness,      // Complete | Incomplete | Unstable
    IReadOnlyList<ReportSource> Sources,  // the revision vector
    PolicySection Policy,                 // PolicyIdentity, redacted rule structure, required evaluator capabilities
    BaselineSection Baseline,             // accepted baseline and catalog identity, non-secret deployment settings
    GitOpsSection Desired,                // desired cluster-gitops revision, per-service chart tag, rendered-settings digest
    IReadOnlyList<ServiceObservation> Applied,  // per instance: image digest, version, supported catalog and contract
                                                // digests, acknowledged PolicyIdentity, readiness; expected count
    IReadOnlyList<DomainPrerequisite> Prerequisites, // e.g. migration version, deletion lifecycle, statistics mode
    IReadOnlyList<string> UnknownOrUnsupported,       // fields this service version cannot report
    string ServiceVersion, string ServiceCapabilitiesDigest);

public sealed record ReportSource(
    string Name,              // "policy", "baseline", "gitops", "workloads", "migrations", "domain:statistics"
    string Revision,          // source's own revision: policy revision, git SHA, observation sequence…
    string Digest,
    DateTimeOffset ObservedAtUtc,
    TimeSpan MaximumAge,
    SourceState State);       // Current | Stale | Missing | ChangedDuringCollection

The report contains no user identities, credentials, secret values or full targeting lists. Secret-sourced settings appear as "from secret" with a digest of the reference, never the value, as the statistics preflight already does.

7.9 Deployment compatibility constraint¶

public sealed record DeploymentCompatibilityConstraint(
    string RolloutId,                     // from the coordinator, e.g. the cluster-gitops promotion PR and SHA
    string EnvironmentId, Guid BindingEpoch,
    IReadOnlyList<string> ServingCatalogDigests,   // every version currently serving
    string TargetCatalogDigest,                    // the rollout target
    CapabilitySet AllowedCapabilities,             // intersection of all of the above
    long PolicyRevisionAtRecord,                   // handshake: the policy revision the coordinator validated
    DateTimeOffset RecordedAtUtc, DateTimeOffset ExpiresAtUtc,
    ActorRef Coordinator);

The constraint is a separate document in the policy collection (_id: "deploymentConstraint", holding the active rolloutId), not part of the policy envelope, so recording, extending or releasing it never changes revision or PolicyDigest and never flips host acknowledgements. Two separate documents would allow write skew (a policy mutation and a constraint write each reading the other's old state and both committing), so both kinds of transaction write one shared guard: every policy mutation and every constraint record, extend or release increments constraintGuard on the policy document (_id: "global"). MongoDB then aborts one of two overlapping transactions with a write conflict, and the retry reads the other's result. constraintGuard is excluded from PolicyDigest, and the policy write path carries it forward when it replaces the document (today's BuildState replaces the whole document, RuntimeFeatureFlagStore.cs:489-496). Policy mutations read the constraint inside their existing transaction (§14.5).

8. The release definition¶

8.1 What counts as released¶

A release is a deliberate production milestone confirmed by the system, not a flag toggle, deployment, visible description or first user visit.

Limited release: intended real use by a selected audience was explicitly released and confirmed. An internal tester enabling a feature for themselves creates no milestone; only a release intent does.
General release: availability to the full intended supported audience was explicitly declared and confirmed. Ordinary permissions and eligibility still apply. It is never inferred from "100 percent" or from a global On.
Public preview publication: a content milestone, recorded separately when it happens at a different time from activation.

8.2 Timestamps¶

Time	Set by	Meaning
Requested	Server clock when the intent is committed	An administrator asked for this milestone
Scheduled	Administrator (optional)	Planned; shown as "Planned" until confirmed
Confirmed	Server, automatically	The first reconciliation at which the activation criteria held: policy committed, every expected host acknowledged a compatible identity, domain readiness complete (§10)
Recorded	Server	When the confirmation event was written; differs from Confirmed only after a crash or outage

If the exact activation time cannot be established after recovery, the milestone records the observed confirmation time with Semantics = ObservedAfterRecovery. It never invents an earlier instant. Administrators never type an actual activation date. A correction is a Corrected event that references the original; the original is never rewritten.

8.3 Time zones¶

Store UTC instants. Detailed history renders exact times in the viewer's time zone with the zone shown. Date-only public headings (What's New, guide) use one declared editorial time zone so the two surfaces never disagree near midnight. The repository convention is UTC: every Quartz schedule sets TimeZoneId = "UTC" (for example ProjectStatisticsMaintenanceSchedule.cs:21) and the statistics charts render UTC. Storage stays UTC. Decided (OD-9, 15 October 2026): date-only public headings use Europe/London, matching UK users' calendar dates; exact times always show their zone.

8.4 Pause, restore and new releases¶

Pausing or stopping appends events and changes the record's status; restoring appends events and keeps the release ID and original milestone dates. Only a genuinely new release (new scope that the intent declares as new) gets a new release ID.

8.5 Historical imports¶

Known past milestones (for example the August 2026 bulk PDF and conditional-question entries in whats-new.json) are imported by a reviewed one-off command with a dry-run diff: Provenance = Imported, Precision = Date where only a date is known, an explicit source, and no announcement. An unknown date stays unknown. An import is never a newly observed activation and never triggers the outbox.

9. Display rules¶

Surface	Production	Staging and previews
What's New	Confirmed milestone date (editorial time zone, date only); unconfirmed items labelled "Planned"; only items the viewer may see	Labelled "Rehearsal"; never a production notification; production history shown only in a separately labelled read-only comparison, if at all
Guide	Public preview or general release date where useful; "Available in this project" from current context	Staging availability with a rehearsal label
Administration	Requested, scheduled, confirmed, recorded, paused and restored events with exact times, zone and status	Same fields, with the environment banner
Private pilot	Milestone visible only to the permitted audience	Synthetic or authorised staging audience only

A user who joins later sees the original date; "New to you" is a separate discovery label. Staging and previews use the same event machinery, but their records live in their own databases under their own binding, and the binding sends staging announcements to a test sink only (§6.3), so a rehearsal cannot create a production record or send a production announcement. Code promotion copies no policy, milestone, date or preference between environments.

10. Propagation, states and freshness¶

10.1 Administrator states¶

State	Meaning
Saved	Committed; no host has acknowledged yet
Applying	Some expected hosts acknowledged; still inside the confirmation window
Active	Every expected serving host acknowledged a compatible policy identity and required domain transitions are complete
Degraded	The window passed with hosts unknown, stale or behind, or a host reports an incompatible catalog
Action required	An operator must act: incompatible binary, binding mismatch, domain transition failed, or authority unavailable beyond its limit

Browsers are never part of Active.

10.2 Expected-host inventory¶

API and PM are the hosts that evaluate runtime policy (README §2). Quartz, Identity, the PDF agent and the S3 notifier do not, and are not in the inventory until they read runtime policy. Expected count per service comes from deployment configuration: a rendered RuntimeFeatureFlags:ExpectedReplicas value per service from the chart's replicaCount. With an autoscaler the inventory reports "observed replicas" and cannot claim Active (OD-17). Each host writes runtimePolicyHostAck on every applied change and every 30 s: instance UID, service (the host's service name), process start ID, image version, supported catalog digest, applied PolicyIdentity, last successful authority check, watch or poll mode, stale state, last error class, and lastSeenAtUtc (when the host last wrote its acknowledgement). {service, lastSeenAtUtc} is the fleet view's index (§6.2, migration 0011). No user or project identifiers.

10.3 What the code does today¶

Path	Mechanism	Interval and limits	Evidence
Legacy API provider (most API gates)	Startup read, then poll	30 s, no jitter; on failure keeps last values with no age limit; equal revision accepted	`RuntimeFeatureFlagProvider.cs:56, 166-199`
Writing API pod	Applies its own commit at once	Immediate on that pod only	`RuntimeFeatureFlagsController.cs:142`
Shared reader (API membership gate, PM statistics serving)	Startup read, then poll; majority read	15 s; 5 s read deadline. Fails closed on the first failed, missing or equal-revision/different-digest read (state `Unavailable` or `Inconsistent`; `Evaluate` returns no decision for any state but `Current`). The 60 s `MaximumAge` applies only when refreshes stop altogether. Healthy observations are at most about 20 s old	`SharedRuntimeFeatureFlagRegistration.cs:62-71`; `SharedRuntimeFeatureFlagReader.cs:45, 56-69, 80, 84-104`; consumers `FeatureGatedAttribute.cs:52-53`, `SharedProjectStatisticsFlagSource.cs:33-46`
Browser	Bootstrap fetch, then poll plus SignalR hint from the writing pod only	10 s bootstrap timeout; 5 min poll; 3 s request timeout, two retries. If the first fetch fails the browser keeps deployed defaults, which is permissive once production can stop a flag whose default is On	`main.ts:284-301`; `runtime-feature-flags.service.ts:32, 86-103`; controller `:143`
Change streams	Not used for policy; infrastructure exists for other collections	n/a	`MongoContext.cs:180-228`; `ChangeFeedSource.cs:69-77`
Fleet acknowledgement	None	n/a	Matrix P3.10

10.4 Assessment of the proposed objectives¶

Decision freshness, 30 s ("new-operation decisions use authority checked within 30 s"). Not met today: the legacy provider has no age limit. The shared reader is already stricter than 30 s in one way (it fails closed on the first failed read) and looser in another (if refreshes stop, it accepts an observation up to 60 s old). Achievable with polling alone, no change streams needed:

Make MaximumAge a per-flag stale limit from the catalog, 30 s for production-eligible flags, enforced at Capture() so the bound holds by construction: a decision either uses an observation younger than 30 s or fails closed (or uses the flag's declared safe state).
Poll every 10 s with ±2 s jitter and keep the 5 s deadline. Load: one single-document majority read per host every 10 s, which is negligible against Atlas.
A change to current behaviour, accepted as a design (OD-3, 15 October 2026) and still to be proved by the freshness measurements: let each flag's catalog entry declare how many consecutive failed reads it tolerates while its last observation is still inside the 30 s limit (proposal: up to two for classified safe consumers, zero for sensitive admission). Tolerating any failed read loosens today's shared reader, which fails closed on the first one, for the two consumers already in scope (disableMembership, statistics serving). Until the measurements are accepted, both keep today's first-failure behaviour.
Move the legacy API provider's eligible consumers onto the shared snapshot, or give the provider the same limit. Without this, the 30 s claim is false for most API gates.

The trade-off is availability: with tolerance, a Mongo blip longer than about 25 s stops admission for eligible flags; without it (today), any single failed read does. That is the intended fail-closed behaviour and must be classified per flag ("Last-known state may continue only for explicitly classified safe consumers").

Enable confirmation, 60 s ("an enable not confirmed within 60 s stays Applying or Degraded and cannot be announced"). Not possible today because nothing records acknowledgements. Achievable once B adds host acknowledgement and the expected inventory: with a 10 s poll, a healthy host applies and acknowledges within about 12 s plus write latency, leaving several retries inside 60 s. A per-pod local SignalR hint on each applied change (P3.11) brings connected browsers on every pod to within the same window, but browsers stay outside the guarantee. Because a browser whose first fetch fails keeps deployed defaults, a flag is production-eligible only if it has a server gate, or if the browser falls back to the catalog's safe state for it (B; RT-14).

Change streams cut typical latency from seconds to well under a second and are what README §4's 2 s p95 needs. They are not needed for 30 s / 60 s. Decided (OD-18, 15 October 2026): ship C on tightened polling; add a watch-then-reconcile wake-up later by reusing the existing change-stream infrastructure, keeping polling as the correctness path.

Measurement.

Unit tests with a fake TimeProvider: an observation is unusable at 30 s plus one tick; acknowledgement timing and the Saved → Applying → Active/Degraded transitions (SyRF.Mongo.Common.Tests).
Hermetic multi-replica run on Bramble (two API, two PM, one browser) in the local E2E stack: 200 commits of an eligible test flag, recording commit time and each host's acknowledgement time; report p50, p95 and max of acknowledgement latency and of observation age at decision time, plus behaviour while one PM replica is partitioned.
Metrics, labelled by service only: syrf_runtime_policy_ack_seconds (histogram) and syrf_runtime_policy_observation_age_seconds (gauge), shown on the fleet view.
Staging operator exercise (human-run, part of C's recovery exercise, with Chris's go-ahead): a short series of toggles of a harmless flag and one emergency stop, reading the fleet view. Automated tests never toggle staging (README §10).

These are targets to trial. They become objectives only when the measurements are accepted (OD-3).

11. Evaluation precedence, grains and targeting¶

11.1 Reconciled order¶

The proposal's order and README §6 agree on most points. The reconciled order, returning a value and a bounded reason:

Constraints. Authorisation, supported binary and catalog, binding and production mode, domain readiness and prerequisites constrain every allow.
Emergency stops force the catalog's declared safe state (not always false), for the environment or an applicable project.
Exclusions (hard deny) dominate every allow. Required beta consent is an eligibility condition here, not a preference a cohort can bypass; opting out of an opt-in beta excludes.
Specific allows, for flags whose grain permits them: combined project-and-user, then user, then investigator, then project. Combined rules need both identities to match. Project-grain flags reject individual rules at save time.
Cohort rules, then the environment override, then the accepted baseline.
Dependencies are re-evaluated in the same context. Cycles, invalid scope combinations and overlapping contradictory rules at one level are rejected at save time.

"Off by default" (baseline Off) and "Stop for everyone" (emergency stop) are different actions in the editor.

11.2 Differences to decide¶

#	Point	Proposal	README §6	Recommendation
P-1	Combined project-and-user rules	Most specific, evaluated first	Not present	Adopt (F)
P-2	Can an individual rule carry Off without being an exclusion?	Not stated; exclusions are the deny mechanism	Step 4 says "explicit user setting" (On or Off)	Allow-only rules plus exclusions; an individual "Off" is written as an exclusion. Removes order ambiguity (OD-13)
P-3	Where consent sits	Eligibility condition at step 3	Step 5, with opt-out excluding	Step 3, as proposed; same outcome, clearer
P-4	Project emergency stop versus project exclusion	"Applicable emergency stops"	"Environment/global or project force-off"	Keep both: a project stop forces the safe state; an exclusion denies the allow. They differ only for flags whose safe state is not Off

11.3 Grains¶

Every flag and feature declares its grain (Global, Project, Personal). Shared project behaviour is evaluated by project, never by the member, so two investigators cannot keep different formats for one project. Personal targeting is only for private UI experiences. In C only Global is used.

12. Docs and announcements¶

12.1 Publication classes¶

Class	Readers	When availability changes
Draft	Authors and reviewers	Stays unpublished
Private pilot	Explicit documentation audience	Membership and permission rechecked on every request; never in a public artifact
Public preview	Anyone	Public explanation stays; the authenticated view explains current availability
Public released	Anyone	Reference and history stay; the authenticated view shows current unavailability
Retired	Audience permitted for the historical content	Archive or redirect with replacement guidance

Temporary disablement does not revoke documentation access; a security or access revocation does. Publication is explicit, never inferred from a flag value. A pilot ending does not make its documentation public.

12.2 Topic IDs, assets and search¶

Topics get permanent IDs (feature-definitions.yaml, TopicIds) and explicit variant and revision references. Flag keys are never topic IDs. A shared link resolves the reader's real permissions and workflow and explains a mismatch. Restricted assets and search snippets get the same checks as articles. Public search indexes hold public material only; protected search runs on the server. The current user guide is a public Jekyll site, so private pilot content has no home until the protected surface exists (OD-15). With OD-4 and OD-15 accepted, private pilot features use in-app guidance only until then.

12.3 What's New¶

The static feed stays for content that ships with a deployment and has no switch, such as fixes and small improvements. Runtime releases get a server projection, written by the outbox (MaterialiseWhatsNew) only after the release is confirmed on every expected host, so it never claims a release that has not been confirmed. Chris decided its behaviour on 16 October 2026 (OD-16):

Where it shows. The home page already shows What's New whether or not the reader is signed in (<app-whats-new /> sits outside the sign-in conditionals at src/services/web/src/app/info/home/home.component.html:84), so personalised entries appear there first. Signed in, the reader sees the public entries plus the entries for features available to them. Signed out, public entries only. The component merges server entries with the static feed in date order. A dedicated signed-in landing page is a later follow-up (§17), not part of D.
A stopped feature never shows as available. While a feature is stopped (normal off or emergency stop), its entry is hidden everywhere, on the public page and for signed-in readers. A restore brings the entry back with its original milestone date (§8.4). Unsent "now available" announcements are cancelled (§12.4); a message already delivered cannot be recalled.
General availability is public. When a release moves from a targeted audience (pilot projects or people) to everyone, its entry appears on the public What's New, dated the day it reached everyone. Until then a targeted entry is never public (§12.1).

12.4 Announcements¶

One announcement per milestone, identified by a stable announcement ID and revision. The outbox key (§7.4) makes retries idempotent. Before any delayed side effect the worker rechecks current policy and release status: a stale enable event after a stop neither re-enables anything nor announces access. A stop cancels unsent "now available" items; an already delivered message is never retracted as if it never existed. A material correction gets a new revision with an explicit re-notification choice. Dismissal uses the discovery store (§7.6).

12.5 What this plan does not claim¶

There is no transaction across pods, browsers, the static site and notification delivery. Intermediate states are explicit: a committed enable with a failed notification stays enabled with the notification pending; a public reveal that fails is recorded as PublicationPending. Required guidance must be ready before a normal enable (OD-5); optional notification delivery never holds a feature back. Emergency stop never depends on docs publication or the outbox. Real announcement delivery is possible only under a production binding; staging delivers to a test sink and explicit test audience.

13. Disable and rollback¶

Emergency stop is a forward policy revision through the emergency permission and the normal propagation path, with no content, release or announcement dependency. Its status lists unconfirmed hosts. Milestone history is kept and current availability shows paused or stopped.
Normal off blocks new use at each consumer's audited boundary. It does not erase data, abort admitted work or discard accepted jobs; each consumer declares whether existing work completes, drains, checkpoints or compensates (statistics, bulk PDF and bulk study update keep their own obligations).
Restore previews a forward change against the current revision and keeps the release ID and original milestone dates.
Binary rollback is separate. The promotion and rollback path refuses an image whose supported catalog and policy schema cannot honour the accepted policy, or that cannot read new data (CD plan G4). The reviewed compatible policy or domain transition comes first. Each host's acknowledgement carries its supported catalog digest, so an incompatible image joining is reported as Action required and receives no gated work.
Provider mode rollback (Shared → Legacy, or Runtime → Legacy in production) requires effective-value parity for every enrolled key, or a reviewed forward revision first (README P3). The check runs in the promotion preflight, never as a pod start-up refusal (RT-15).

14. Environment-state and feature-policy service and deployment checks¶

From Chris's revised proposal and service addendum (2026-10-15), which supersede CD plan D1 option A. The design below is the repository's reading of it against the code; schema details, intervals and retention figures are proposals.

14.1 One logical boundary, four scoped views¶

The service is the existing API and flag infrastructure extended, not a new deployable or one large endpoint. Every view uses the same catalog, the same evaluator (B's shared evaluator) and the same authoritative policy document. The admin UI keeps editing policy through the existing revision-checked, audited transaction; there is no second configuration system. Deployment-bound values stay read-only in the UI.

Caller	View	Routes (§7)	Authorisation
Application and guide	Evaluated availability for the authenticated user, project and workflow, with freshness and bounded reasons	`GET /api/features/availability`; the existing anonymous snapshot for the public subset	Signed-in user; membership checked on the server
Flag administration	Catalog, policy, deployment constraints, desired and applied state, fleet observations, preview and commit, emergency stop, audit	`/api/runtime-feature-flags/`, `/api/feature-releases/`	Per operation (OD-2); real actor under impersonation
Deployment automation	Read-only, versioned compatibility report; environment binding; relevant policy and configuration; deployed capabilities; validation status	`/api/environment/compatibility-reports/**`; constraint routes for the coordinator only	Environment-bound machine identity with a read-only scope; the coordinator scope is separate (OD-21, OD-22)
Internal collectors	Narrow publication of observed workload, version and readiness facts	`POST /api/environment/observations`	Collector scope; no read of policy, no mutation of policy

14.2 Where the facts come from¶

Fact	Source today	Gap
Accepted policy identity and structure	`runtimeFeatureFlagOverrides` (legacy shape, matrix P3.2)	Envelope and digests (B)
Accepted baseline and catalog identity	Per-pod configuration (matrix P4.1)	Baseline import (B)
Desired GitOps revision and rendered-settings provenance	cluster-gitops files; the statistics staging preflight already pins and compares them (`scripts/capture-statistics-preflight.py`, statistics STATUS)	A collector that records the desired revision per service; the ArgoCD Application status is the natural source
Per-instance image, version, readiness	`syrf_metadata` instance records (`DatabaseMetadataWriter.cs:160-168`); in staging the read-only `syrf-stats-preflight` Kubernetes identity reads Deployments and pod image digests (#3806)	A collector per environment publishing to `runtimeWorkloadObservation`; no production equivalent of the staging Kubernetes reader exists yet (CD plan D5)
Applied policy per host	None (matrix P3.10)	`runtimePolicyHostAck` (B)
Domain prerequisites	Migration state (`syrf_migrations`), deployment flags such as `deletionLifecycle`, statistics modes	Report section listing each, unknown when unreadable

An API replica cannot infer the fleet from its own version; it reports what the collectors and heartbeats published and how old each fact is. "All information" means the defined report contract, never an environment-variable dump or user data.

14.3 Report consistency¶

The sources share no atomic snapshot. The service:

records each source's revision and digest at the start of collection (the revision vector);
reads the sections;
re-reads each source's revision at the end;
marks the report Complete only if every source is Current and unchanged; otherwise it recollects up to three times (a proposal), then returns Unstable (something changed) or Incomplete (something missing or stale).

A report ID alone does not establish consistency. Validators treat Incomplete, Unstable, an expired report and any field in UnknownOrUnsupported that the check needs as a failure, never as ready. Retained reports (runtimeEnvironmentReport) exist to reproduce a check; live authority stays in the originating stores.

14.4 Authentication prerequisites (verified)¶

Environment	What exists	Evidence
Staging	SyRF Identity (OpenIddict) deployed; the API accepts OpenIddict access tokens; client-credentials flow enabled; precedent machine clients: `syrf-statistics-evidence`, whose single scope `statistics:parity:read` the API accepts on one route only, and the statistics operator credential `statistics:operate`	cluster-gitops `syrf/environments/staging/identity/config.yaml` (`enabled: true`) and `staging/api/values.yaml` (`identityProvider: openiddict`); `Identity.Endpoint/Program.cs:142`; `StatisticsParityEvidenceClient.cs:4-23`; API `Program.cs:481-485`
Production	No SyRF Identity (`service.enabled: false`) and no OpenIddict validation configured in production's values, so only the Auth0 JWT scheme (registered in every environment, API `Program.cs:589-593`) can authenticate there. That scheme would authenticate an Auth0 machine-to-machine token issued for the API audience; what is missing is an authorisation path for a machine principal, which today exists only for the staging OpenIddict clients. Whether the Auth0 tenant has suitable grants cannot be verified from the repository. The Auth0 Management API client is outbound (API to Auth0)	cluster-gitops `syrf/environments/production/identity/config.yaml` and production API values (no OpenIddict settings); `AuthManagementApiClientProvider.cs:39`; auth migration status

So the automation identity cannot assume OpenIddict in production. Decided (OD-21, 15 October 2026): staging uses an OpenIddict client modelled on syrf-statistics-evidence with the scope environment:report:read; production reuses the approved current provider, Auth0, with a dedicated machine application and the same scope name (an Auth0 tenant change, which needs Chris's approval because the tenant is production-shared), until the Identity migration reaches production. Both scopes are accepted on the report routes only, the token's environment audience must match the binding, and the credential lives in a main-only GitHub environment secret, as the statistics evidence credential does. Authenticated never means unrestricted: the actor and target environment are checked on every call.

14.5 Concurrent deployment compatibility¶

A preflight alone leaves a race: an administrator could enable a capability the rollout target lacks after the preflight passed. The policy write path closes it:

Before a rollout, the trusted coordinator (the promotion workflow) fetches a fresh report and records a DeploymentCompatibilityConstraint (§7.9) naming every serving catalog digest and the target's, with the policy revision it validated. The constraint write is a compare-and-set in one transaction: it fails if the current policy revision differs from PolicyRevisionAtRecord, and it fails if the current accepted policy uses any capability outside AllowedCapabilities. Because the constraint write and every policy mutation both increment the shared constraintGuard (§7.9), they cannot both commit while overlapping: an enable that commits first makes the constraint write fail its revision check, and an enable that starts after the constraint commits sees it and is rejected. In either case the coordinator starts again from a fresh report. The coordinator's re-read of the policy revision after recording (step 3) is a second check, not the only one.
Every policy mutation, in its existing transaction, reads the active constraint and rejects (or defers to after the rollout) an enable or baseline change outside AllowedCapabilities. Compatible changes and emergency stops are always allowed; administration is not frozen.
After recording, the coordinator re-reads the policy revision before each rollout step; if it moved, the coordinator revalidates against the new policy before continuing. Expiry fails closed: a constraint is lifted only by the coordinator's release (after the collector observes every replica on the target, or after a completed rollback). An expired but unreleased constraint keeps blocking incompatible enables, is reported as Action required, and can be extended by the coordinator.
Readiness blocks a new replica whose supported catalog cannot honour the accepted policy (§13).
The service upgrades itself through versioned expand/contract changes; the old service must report enough for the new validator to check the transition (report schema N and N-1 both accepted).
Bootstrap. The first report-capable version cannot be validated by a report from the version it replaces, because production's current build (104eca9) has no report endpoint. The catch-up promotion and the first deploy of the report endpoint are validated by the CD plan's legacy checks (G2 on the interim cluster-gitops input, D16, and G4) until a report-capable version serves production; the report-based lanes take over from the next promotion.

Candidate tests still supply the evidence that new code is correct; the report only describes the environment. Rollback separately validates data and policy compatibility (CD plan G4).

14.6 The same workflow in staging and production¶

The trusted pipeline authenticates to staging's service and confirms the environment identity against the binding.
It obtains a staging report, validates the candidate, and runs deployment, post-deployment checks, runtime enable and disable, and docs and announcement rehearsal (announcements to a test sink).
It exercises failure paths: stale or missing observations, authentication failure, concurrent policy edits, report schema transition, mixed versions and rollback.
The same validated artifact and checker version go to production's workflow, which fetches its own fresh report and repeats the applicable checks against production state. A staging pass is evidence about the mechanism, not authorisation.
Relevant state is revalidated immediately before deployment and compatibility and readiness are monitored throughout the rollout.

Addresses, credentials, databases, policy histories and notification delivery are isolated per environment; the endpoints are structurally the same. No staging action can write a production milestone (§6.3 binding).

14.7 CI lanes¶

These replace the earlier lane table and the CD plan's vendored-copy inputs. Each lane is defined by its input and what a failure means.

Lane	Input	Failure means	Where it runs	Credentials
PR policy and contract tests	Synthetic, versioned fixtures in the repository; evaluator and API contract tests	Wrong rules, permissions, dependencies, safe operation or contract compatibility	Every PR, including forks (`test-dotnet`, web specs)	None
Trusted candidate compatibility	An immutable report for the target environment, fetched by a separate trusted collection job; candidate and base renders; policy structure; supported capabilities	The candidate cannot preserve currently enabled behaviour or understand accepted configuration or policy	Same-repository heads; the collection job never runs candidate code	Collection job only, read scope; the candidate-evaluating job receives the sanitized report as an artifact
Deployment preflight	A fresh report plus revalidation of the relevant revisions and capability floor, then the constraint (§14.5)	Earlier evidence is stale, incomplete or incompatible	Promotion workflow, main only	Read scope plus coordinator scope
Runtime health	Accepted policy versus each required consumer's acknowledged state	Propagation, stale-state or lifecycle failure; not a Git or default mismatch	Scheduled and during rollout	Read scope
Staging rehearsal	The same contracts, credential flow and checks; staging state plus production-like synthetic fixtures	Broken check workflow, missing coverage or failure handling; never authorises production	Staging promotion	Staging credentials only

Until the authenticated report exists for an environment, the CD plan's G2 uses an interim input (its D16, a decision for Chris): a trusted job renders production's chart tags from cluster-gitops at one resolved SHA for that run only, keeping the result as an access-controlled run artifact and committing nothing. Before production runtime cutover, legacy checks (the CD plan's render comparison) stay only where their assumptions hold (production in Legacy mode). The authenticated report is introduced incrementally; an unsupported field is reported as unknown, never treated as ready. An anonymous snapshot cannot establish targeted policy or fleet convergence. Untrusted PR jobs never run candidate code with a service credential, and nothing writes production identities, credentials or full targeting lists to the repository or public artifacts.

15. Acceptance and failure tests¶

Each case names its layer and verification. Test projects: SyRF.API.Endpoint.Tests, SyRF.Mongo.Common.Tests, SyRF.ProjectManagement.Mongo.Data.Tests (replica-set fixture), SyRF.ProjectManagement.Endpoint.Tests, web specs via pnpm exec ng test, Playwright specs in e2e/tests/, and helm unittest in the API and PM charts. "Hermetic multi-replica" means the local E2E stack on Bramble with two API and two PM processes.

#	Case	Layer	Verification
RT-01	Two editors at the same revision: one commits, the other gets 409 and keeps its draft	Store integration tests (replica set): `SyRF.Mongo.Common.Tests` if B moves the store into the shared library with the evaluator (OD-24, accepted), otherwise `SyRF.API.Endpoint.Tests`; web spec for the draft	Test run; spec asserts the draft survives
RT-02	Retry after an unknown commit with the same mutation ID returns the original result: one state change, one audit row; same ID with a different payload is rejected	Store integration tests (as RT-01)	Counts state revisions and audit rows
RT-03	Cold start with authority unavailable: eligible risky paths fail closed; no permissive default re-enables a stopped flag	`SyRF.Mongo.Common.Tests`, `SyRF.API.Endpoint.Tests`	Decision is null/safe state; route returns typed unavailable
RT-04	Missed notification, invalid resume token (if watch added), revision regression and equal-revision/different-digest each leave a stopped flag stopped	`SyRF.Mongo.Common.Tests`	Observation states `Inconsistent`/`Stale`; decision unchanged
RT-04a	A single failed read inside the age limit behaves as the accepted policy says: no decision (today's first-failure rule) unless the flag's catalog entry declares tolerance and OD-3 accepted it	`SyRF.Mongo.Common.Tests` (fake `TimeProvider`, failing reader)	Decision per configuration
RT-05	Epoch change without an acknowledged rebind quarantines the reader	`SyRF.Mongo.Common.Tests`	State `Quarantined`; no decision from the new lineage
RT-06	Restore drill: restored copy fails the binding check until `rebind`; rebind rotates `bindingEpoch` and sets announcement delivery to `disabled`	`SyRF.ProjectManagement.Mongo.Data.Tests` + operator exercise	Exit code 2 before rebind; record fields after
RT-07	Binding check refuses a wrong database name, wrong cluster fingerprint, wrong `RuntimeEnvironment` and a missing record, writing nothing	`SyRF.Mongo.Common.Tests` (`Migrations/`)	Exit code 2; `syrf_migrations` unchanged
RT-08	Release-store migrations are idempotent, adopt identical indexes, pass the lock-file test and ADR-023's target-3 rollback rule	`SyRF.ProjectManagement.Mongo.Data.Tests`	`PmMigrationsLockFileTests`; Up, Down, Up; for `0011`–`0020`, `PmRuntimeReleaseIndexMigrationsTests` (replica set)
RT-09	Browser: bootstrap revision seeds the service; an older poll response cannot overwrite; reconnect triggers a fetch; revision parsed from a decimal string	Web spec `runtime-feature-flags.service.spec.ts`	`pnpm exec ng test --include=…`
RT-10	Production mode `Legacy` ignores stored policy exactly as today; `Runtime` honours it only for production-eligible keys and ignores stored overrides for every other key; switching needs the deployment value only	`SyRF.API.Endpoint.Tests` + `helm unittest`	Parity test over the catalog; per-key eligibility test; chart render
RT-11	Two API, two PM and a browser converge to Active; a silent replica is unknown and the state is Degraded at the accepted confirmation window (trial 60 s); a new replica with an older catalog digest is Action required and gets no gated work	Hermetic multi-replica	Fleet endpoint output captured; measurements (§10.4)
RT-12	Emergency stop reaches all hosts; new decisions use it within the stale limit; it succeeds with the outbox, docs and notification paths unavailable	Hermetic multi-replica	Decision log timestamps; outbox stopped during the run
RT-13	A non-eligible flag cannot be changed in production; capabilities are enforced separately (policy change, emergency stop, inspect); under each impersonation path the test asserts which identity is authorised (today the impersonated target's authority, with no escalation) and that the audit records both real and impersonated identities	`SyRF.API.Endpoint.Tests`	403 per capability; identity asserted per path; audit fields
RT-14	A browser whose first snapshot fetch fails never shows a production-eligible flag as On when its policy is stopped: either the flag has a server gate that refuses, or the browser falls back to the catalog's safe state	Web spec + `SyRF.API.Endpoint.Tests`	Spec with a failing fetch; route refusal
RT-15	Switching production from `Runtime` back to `Legacy` is refused by the promotion preflight, before the cluster-gitops change merges, while an emergency stop is active or any enrolled key's legacy value differs from the runtime decision, until a reviewed forward revision makes them agree. Pods never refuse to start over this: after the catch-up promotion API and PM take the chart's `Recreate` default unless production overrides it (CD plan A7, D10), so a start-up refusal would take every pod down, and even under `RollingUpdate` it would stall the rollout	Preflight checker tests + `SyRF.API.Endpoint.Tests` for the parity computation	Preflight fails with the differing keys; pods start normally in either mode
RT-20	Release intent commit: policy, intent, `PolicyCommitted` event and outbox rows in one transaction; a forced failure leaves none	`SyRF.Mongo.Common.Tests` or API data tests (replica set)	Collections inspected after injected failure
RT-21	Confirmation is derived only when every expected host acknowledged; `ConfirmedAtUtc` comes from the server clock	`SyRF.API.Endpoint.Tests` with fake `TimeProvider`	Milestone fields asserted
RT-22	A crash at each boundary (after intent, after commit, after confirmation, mid-outbox) resumes to the same end state with one event per milestone and one announcement	API data tests (replica set)	Fault injection at each step
RT-23	A delayed enable or announcement processed after a stop neither re-enables nor announces	API data tests	Outbox guard rejects; `AnnouncementCancelled` recorded
RT-24	Stop then restore keeps the release ID, original milestone dates and announcement identity	API data tests	Event and record assertions
RT-25	A staging or preview binding cannot dispatch outbound announcements; a staging release never appears in production	API data tests + binding tests	Outbox dispatcher refuses; separate databases
RT-26	Normal release enable is refused without required guidance or with an incompatible host; emergency stop still works	`SyRF.API.Endpoint.Tests`	422 with reason; stop succeeds
RT-27	Historical import keeps precision and provenance and queues no outbox items	API data tests	Event fields; outbox empty
RT-28	Public build, public search index and public assets contain no private-pilot content; direct links and old revisions are checked	Content build test (H) + API tests	Build artefact scan; 403 on direct link
RT-29	What's New projection shows confirmed milestones in the editorial time zone and "Planned" for scheduled ones, including a milestone at 23:30 UTC	Web spec for the component + API tests	Rendered date asserted
RT-30	Project rule enables only that project; another project is unaffected; dependency resolution never widens a parent to all projects	`SyRF.API.Endpoint.Tests`, PM tests	Truth table
RT-31	Shared project behaviour evaluates by project for every member	PM tests	Two members, one decision
RT-32	Project switch and permission revocation invalidate feature and guide decisions; a late response for the old project is discarded	Web spec + Playwright	`contextId` mismatch discards
RT-33	Combined project-and-user rule, exclusions, investigator mapping, opt-out and unsupported scopes give the documented truth table	`SyRF.API.Endpoint.Tests`	Table-driven test from the proposal's example
RT-34	Ambiguous user-to-investigator mapping never matches a targeted enable	`SyRF.API.Endpoint.Tests`	Decision is baseline
RT-35	Revoked membership drops personal decisions even when policy is unchanged	API tests + web spec	Next evaluation unavailable
RT-36	Cohort hashing matches committed cross-language vectors; growing from 5 % to 25 % keeps the earlier cohort	`SyRF.Mongo.Common.Tests`	Golden vectors
RT-37	Missing stable identity never falls back to random assignment	`SyRF.Mongo.Common.Tests`	Decision is baseline
RT-40	Docs outage (content service and static site unreachable) does not delay an emergency stop	Hermetic multi-replica	Stop timing unchanged
RT-41	Discovery: dismissing an announcement hides it for that user only; a later joiner sees the original date with "New to you"	API tests + web spec	Per-user state
RT-50	Binary rollback or a promotion candidate that cannot honour accepted policy or read new data is refused	CD plan G4 lanes + deployment preflight (§14.7)	Lane fails on a fixture
RT-51	Disable drains or preserves accepted work per consumer contract; statistics and bulk-job invariants hold	Existing domain test suites per consumer	Named tests in each consumer PR
RT-52	A runtime toggle changes no PR fixture result and needs no repository update	PR policy and contract lane	Lane output stable across toggles
RT-60	Each scoped view returns only its fields: availability exposes no policy structure; the automation view exposes no identities, secret values or full targeting lists; the collector scope cannot read or change policy	`SyRF.API.Endpoint.Tests`	Contract snapshots per view; 403 for cross-scope calls
RT-61	A source that changes between the start and end of collection makes the report `Unstable` after the bounded retries; a missing or stale source makes it `Incomplete`; validators fail both	`SyRF.API.Endpoint.Tests` with fake sources	Report state asserted; checker exit code
RT-62	The same checker version passes against a staging report and is rerun against a fresh production-like report; a reused staging report is rejected for production (environment and binding mismatch)	Checker unit tests + hermetic run	Checker refuses a foreign `environmentId`/`bindingEpoch`
RT-63	An expired report, a report with schema N-2, or a needed field in `UnknownOrUnsupported` fails the check; schema N-1 is accepted during an upgrade	Checker unit tests	Fixture reports
RT-64	A token for one environment's audience is refused by the other environment's service; a read-scope token cannot call the constraint or observation routes	`SyRF.API.Endpoint.Tests` (OpenIddict and Auth0 schemes)	401/403 per case
RT-65	The trusted collection job never executes candidate code; the fork lane runs with no service credential	`.github/scripts/validate-workflows.sh` contract + workflow review	Validator fixture fails when a credential reaches a candidate-running job
RT-66	With an active deployment constraint: an enable outside the capability intersection is rejected; a compatible change and an emergency stop succeed; a constraint write fails if an enable landed after the coordinator's report (revision moved, or accepted policy outside the intersection); an expired but unreleased constraint still blocks incompatible enables; interleaved: a policy enable and a constraint write started concurrently, each before the other commits, never both commit (one aborts with a write conflict, and its retry is rejected or fails the revision check); a break-glass release is audited and admin-only	`SyRF.Mongo.Common.Tests` (replica set)	Transaction outcomes per case
RT-67	During a mixed-version rollout a new replica whose catalog cannot honour accepted policy stays not ready and receives no gated work	Hermetic multi-replica	Readiness and fleet view captured
RT-68	What's New (OD-16): a stopped feature's entry is hidden on the public and signed-in views; a restore shows it again with its original date; reaching everyone publishes it publicly, dated that day; signed out, the home page shows public entries only	Web spec for the component + API tests for the projection	Rendered entries per viewer and state
RT-69	Release rehearsal (OD-26): switching a release on in staging makes its guide page and What's New entry available to its audience at the same moment; switching it off hides the What's New entry and the guide shows the feature as currently unavailable (§12.1). The same check passes before the production enable	Staging rehearsal checklist, then the production pre-enable check	Screenshots or captured responses before and after each switch, recorded with the release

16. Owner decisions for Chris¶

Chris answered every decision through the runtime releases decision sheet on 15 and 16 October 2026. "Accepted" means the recommendation stands as written. Answering a decision does not authorise any production change (§1).

#	Decision	Recommendation	Chris's answer
OD-1	First production runtime control	`disableMembership`: already on the shared reader in staging, with an audited admission boundary, reversible, admin-facing. Trade-off: its only server consumer is the API route `ProjectController.cs:1217`; PM would acknowledge policy but never consume it, so it does not "genuinely exercise API/PM" (README §11 decision 3). Recommendation: pair it with the PM-consumed statistics-serving stop where statistics are enabled, or accept an API-only first control and prove PM consumption in staging. First user-visible release candidate for D: `screeningKeywordHighlighting`. Never statistics maintenance	Accepted, API-only (15 Oct): `disableMembership` first; Project Management consumption is proven in staging. Production's build (`104eca9`) has no materialised-statistics code (statistics status), so the pairing is not available yet
OD-2	Production permission mapping	Split `ManageRuntimeFeatureFlags` into change policy, emergency stop, inspect targeting and manage releases; all mapped to Administrator at first; real actor recorded under impersonation	Accepted (15 Oct)
OD-3	Freshness and convergence objectives	Trial 30 s decision freshness and 60 s enable confirmation on 10 s polling (§10.4); accept only after the measurements. Includes the change of whether a flag may tolerate failed reads inside its age limit (today's shared reader fails closed on the first one); recommend per-flag, zero for sensitive admission	Accepted (15 Oct)
OD-4	Confidential versus public pilot content	Classify per feature and release; default unfinished material to Draft or Private pilot	Accepted (15 Oct)
OD-5	Required guidance before release	Required for features that change user tasks; reviewed exemptions for invisible operational controls	Accepted (15 Oct)
OD-6	Sensitive audit retention	Reuse the approved retention and access policy; no identities in metrics or Git; settle subject deletion before personal targeting (F)	Accepted (15 Oct)
OD-7	Public renderer migration	Keep the provisional Starlight recommendation, subject to the design work and a representative migration proof	Accepted (15 Oct)
OD-8	How production policy is switched on	Deployment-only `RuntimeFeatureFlags:ProductionPolicyMode=Legacy\|Runtime`, default `Legacy`, changed by a Chris-approved cluster-gitops PR (ADR-025)	Accepted (15 Oct)
OD-9	Editorial display time zone for date-only headings	Storage is UTC in every case (the repository convention). Display: UTC (convention) or Europe/London (proposal). Recommend Europe/London for public headings only, because users read calendar dates in UK time, with exact times always shown with their zone	Accepted (15 Oct): Europe/London for public date-only headings; storage stays UTC
OD-10	Migration placement and validators	Append to `PmMigrations` from `0009` as index migrations on migration-only collections; no `$jsonSchema` validators in the first slice. The two audit indexes (`0009`–`0010`) touch the existing `runtimeFeatureFlagAudit` collection: decide between adding `runtimeFeatureFlagAudit` to `MigrationOnlyCollections` (it meets the rule's definition) and moving ADR-023's rollback target forward after the catch-up promotion (§6.4); the catalogue is append-only, so reordering is not possible	Option C (16 Oct, Chris's suggestion): an undo leaves an index it found already in place (`adopted`), so the rollback rule becomes "every migration after version 4 is an index migration" and `0009`–`0010` need no special treatment. Runner change and ADR-023 amendment in #4255, before `0009`. A future drop-index migration's undo recreates the index. No `$jsonSchema` validators in the first slice (§6.4)
OD-11	Previews	Land the pre-deploy plan's preview `up` item before D; until then release features are unavailable in previews	Accepted (15 Oct)
OD-12	Binding bootstrap	Production and staging `bind` run once each through a reviewed cluster-gitops Job values change; previews and local bind in their lifecycle Job	Accepted (15 Oct)
OD-13	Precedence differences P-1 to P-4 (§11.2)	Adopt the recommendations in §11.2	Accepted (15 Oct)
OD-14	Discovery preference store	A new per-user collection; `ReviewerWorkspaceSettings` is review-workspace scoped, and the email preferences in open #3942 are a different concern	Accepted (15 Oct)
OD-15	Protected documentation surface	Serve private-pilot and authenticated content from the API into the app; keep the public site public; no new hosted docs service	Accepted (15 Oct). A wider help-guide overhaul, possibly on a different framework, is a separate pass (§17)
OD-16	What's New for runtime releases	Server projection merged with the static feed; static feed kept for deployment-time content	Accepted with changes (16 Oct): a stopped feature's entry is hidden everywhere until restored, keeping its original date; reaching everyone makes the entry public, dated that day; personalised entries appear first on the existing home page, which already shows What's New when signed in, with a signed-in landing page as a follow-up (§12.3)
OD-17	Expected-host inventory source	Rendered expected replica count per service; "observed replicas" only, with no Active, when an autoscaler is used	Accepted (15 Oct)
OD-18	Change streams	Not for C; add later as a wake-up with polling kept as the correctness path	Accepted (15 Oct)
OD-19	Announcement delivery outside production	Staging: a test sink and explicit test audience; previews and local: disabled; restored copies: disabled until rebound	Accepted (15 Oct)
OD-20	Report freshness and retention	Report expiry 15 minutes after collection; recollect at most three times before `Unstable`; retain reports 30 days; review after the first staging rehearsals	Accepted (15 Oct)
OD-21	Automation identity per environment	Staging: an OpenIddict client modelled on `syrf-statistics-evidence`, scope `environment:report:read`. Production: until Identity reaches production, an Auth0 machine application with the same scope name (production-shared tenant change, needs Chris's approval). Main-only GitHub environment secrets; report routes only (§14.4)	Accepted (15 Oct)
OD-22	Deployment coordinator and constraint	The promotion workflow records the constraint with a separate coordinator scope. One active constraint per environment. Constraints never lapse on their own: after their expiry time they keep blocking incompatible enables and are reported as Action required until the coordinator extends or releases them. If the workflow dies, an administrator uses the audited break-glass release. Emergency stop is never blocked (§14.5)	Accepted (15 Oct)
OD-23	Workload collector	One collector per environment publishing Deployment and pod facts with a read-only Kubernetes identity (staging has one, `syrf-stats-preflight`; production needs a camarades-infrastructure change, CD plan D5)	Accepted (15 Oct)
OD-24	Home of `RuntimeFeatureFlagStore`	Move it into `SyRF.Mongo.Common` with the shared evaluator in B, so PM and tests share one write path; the API keeps the controller	Accepted (15 Oct)
OD-25	Release reconciler and outbox dispatcher host (review question 2)	One leased worker per environment. The closest precedent is the deletion runner, which runs in Project Management on a Quartz recurring schedule (`DeletionLifecycleSchedule.cs`) and leases each operation it advances (`pmDeletionOperation` `Lease.ExpiresAtUtc`, migration `0006`); recommend the same shape in PM, with a lease per release or outbox item. `ConfirmedAtUtc` comes from the lease holder's clock and the event records which host set it	Accepted (15 Oct)
OD-26	Whether a first production enable of `disableMembership` is a user-visible release (review question 3)	It is Off in production by default. Production's current build has the membership model but not the `disableMembership` flag or its route gate. Treat its first enable as a limited release for project administrators, needing guidance under OD-5	Accepted with an addition (16 Oct): a real feature release, sequenced as staging switch-on tested by Chris, then the help guide, then production for project administrators once Chris approves. The staging switch-on is also the test that the feature's guide page and What's New entry become available at the same moment as the switch; the same check runs before the production enable (RT-69)
OD-27	Production-history comparison in staging (review question 4)	Rule it out until a reviewed read-only cross-environment view exists; staging holds no production credential	Accepted (15 Oct)

Open questions¶

#	Question	Recommendation	Chris's answer
Q1	Should the emergency stop bypass index readiness, if a new audit index is missing?	Recommend yes: the emergency stop writes through the existing state and audit path and must not depend on the `0009`/`0010` indexes; a binding mismatch still refuses it	Accepted (16 Oct): the emergency stop always works; a binding mismatch still refuses it
Q2	Which process runs the reconciler and outbox, and whose clock confirms?	OD-25	Answered by OD-25
Q3	Is `disableMembership` a user-visible release?	OD-26	Answered by OD-26
Q4	Optional production-history comparison in staging?	OD-27	Answered by OD-27
Q5	If the emergency stop bypasses index readiness (Q1) while the audit `{mutationId}` unique index is still missing, can a retried stop write a duplicate audit row?	Yes: without the unique index nothing stops a second insert. The policy state stays correct (the retry checks the expected revision and, if the first attempt committed, reads it back by mutation ID before writing), so the risk is a duplicate audit row only. Mitigation: look up the mutation ID inside the transaction before inserting, and treat a duplicate as idempotent once the index exists	Accepted (15 Oct)

17. Follow-ups found while writing (not part of this plan's deliverable)¶

A wider help-guide overhaul, possibly on a different documentation framework (compare OD-7's renderer recommendation), as its own pass (OD-15 note, 16 October 2026).
A dedicated signed-in landing page for personalised What's New; until then the existing home page shows it (OD-16).
Update the What's New feed describes a latest/past feed shape; the feed and schema use an updates array. Fix in a small docs PR.
The operating guide generator example still prints ${SPA__MyNewFeature} while stating that the SPA__ prefix is legacy.
MongoDB reference does not list runtimeFeatureFlagOverrides or runtimeFeatureFlagAudit; the implementation PRs add those and the new collections.
Keep binding, policy, release and report collections out of the preview snapshot allow-list (src/charts/preview-infrastructure/values.yaml, databaseLifecycle.snapshot.collections), because the snapshot producer copies production collections onto the Preview cluster.
SYRF__RuntimeEnvironment defaulting to the mutable local is tracked in #4231.