Continuous delivery and promotability guardrails plan (October 2026)

This is a plan only. Nothing in it is implemented by this PR.

Chris set the priority on 2026-10-14:

"should be a priority to not break production promotion flow in future by breaking encapsulation of partial implementation work that should be isolated behind feature flags. main should always be ready to promote to prod. (with the eventual goal is continuous delivery but even more importantly nearer term goal of main always being ready for a future promotion with minimal extra steps for testing)"

He has approved adding guardrails that protect promotability. This plan designs them (Phase 1), then describes how promotion becomes a regular, small event (Phase 2) and finally continuous delivery (Phase 3). The specifics in each phase still need Chris's review; the open decisions are in §8.

Update 2026-10-15. Chris refined D1: deployment checks read an authenticated environment-state and feature-policy service (scoped views; the same contract and check workflow in staging and production, with separate state and credentials), superseding option A, the vendored repository copy of production values. The service, its compatibility report and the post-cutover lanes are designed in the runtime feature releases plan §14. G2, G7, G8 and G9 below now read that report instead of a vendored copy. Checks that assume "production ignores runtime overrides" hold only while production runs the legacy policy mode (ADR-025, Proposed).

Continuous deployment is not the goal. Production deployment always needs Chris's explicit approval: the cluster-gitops production PR is merged by an administrator after a code-owner approval. Nothing in this plan weakens that. If Chris ever wants automatic production deployment, that is a separate, later decision.

1. Why: the 2026-10-14 promotion audit

Production runs builds from August. API and PM are at 104eca9 (the 2026-08-05 merge-base 6c9b72e2f plus four restoration commits), web is at web-v7.39.0 (6c9b72e2f) and quartz at quartz-v0.35.1 (2026-03-20). Main is about 7,100 commits and 2,276 merges ahead. A read-only audit on 2026-10-14 (syrf 19c5797af, cluster-gitops c04bdecc) rendered main's charts with production values and found that main cannot be promoted as it is. The findings that matter for this plan:

#	Finding	Why it broke promotability	Introduced by
A1	Seven web flags that are ON in production would switch OFF. The API's runtime-flag snapshot is now authoritative in the browser, the API reads FeatureFlags:<Prop> ?? false, and production's API values carry none of the keys that production's web values turn on (contactUsForm, newStageOverview, newScreeningOverview, newProgressIndicators, quantitativeDataExportEnabled, showUpdateInclusionInfo, logRocket)	A change to where a flag's value comes from silently changed the effective production value	#3030, commit be28f8c934 (2026-09-01): applyRuntimeFeatureFlagSnapshot switched from "web deployed value plus overrides" to snapshot.values[key] for every catalogued key. The overrides came in with #2943
A2	Project, search and import-job DELETE return 503 unconditionally, whatever deletionLifecycle is set to. Resolved by syrf#4172 (owner decision 2026-10-14): flag off restores production's immediate deletion; flag on keeps the 503 until the reversible lifecycle lands	Existing production behaviour was removed while its replacement is unfinished, with no flag	#2843, last touched by #3159
A3	SessionTally gains four always-written fields that 104eca9 cannot read. In 104eca9, Entity has [BsonExtraElements] but nested value objects such as SessionTally have no extra-elements handling and there is no global convention, so deserialisation throws FormatException	Blocks rollback, and breaks old pods during any old/new overlap	#2467
A4	Unconditional index builds at startup on pmStudy (about 10.7 GB) through the IRunAtInit hook CreateIndexesAndMapping, before Kestrel listens. Liveness has a 60 s initial delay and there is no startup probe	A long build means crash-looping pods during promotion	#2991, #3909, #2788
A5	The s3-notifier Lambda (≥1.19.0) throws on a null versionId. Production's syrfapp-uploads bucket is not versioned; staging's is, so staging cannot show the failure	A code assumption about the environment that only production violates	#2841
A6	PM renders SYRF__FeatureFlags__BulkStudyUpdateRequireCurrentExecutionVersion twice (false from the new chart default, true from production's raw env:). Kubelet takes the last value; a sync could collapse it the wrong way	A new chart default collided with an environment override	#3387. Cleared on main by #4155 once merged
A7	deploymentStrategy: Recreate becomes the chart default for API and PM. Production runs RollingUpdate today and does not override it, so every production deploy becomes a full outage	A chart default change that only production values would expose	#2976
A8	Authorization tightenings in force regardless of flags (data export, study reads, full stats, membership reads)	Intended security fixes, but user-visible in production with no announcement trail	#3243, #3875, #3892, #3340, #2271
A9	Inert production values go live with the new chart: JSON console logs, Prometheus PodMonitoring on port 9464	Production values that nobody had seen rendered	#3343/#3515, #3597
A10	featureFlags.apm: false cannot switch APM off: the template's default "true" swallows false	A kill switch that does not work	env-mapping template. Cleared on main by #4155 once merged
A11	RIS import is ungated in the web and API, but the production Lambda (1.17.1) cannot parse Ris	A shared enum value reached a separately deployed component that predates it	#2971
A12	Staging does not exercise what production users get: newScreeningOverview and newProgressIndicators are effectively OFF on staging but ON in production; other staging "ON" states exist only as Mongo runtime overrides, which production ignores	Staging testing does not cover the production configuration	cluster-gitops staging values + runtime overrides
A13	Unflagged behaviour changes on hot paths: a new 409 AlreadySufficientlyScreened on screening save; producers writing delegated-work admissions with enforcement off; ungated endpoints behind web-only gates	Partial work leaking past its flag	#3579, #3898

Every finding has the same root cause: nothing checks main against production's configuration and data until someone attempts a promotion. Each PR was reviewed and tested; none was tested against "production values, production data shape, the production build still running alongside it".

The six rules Chris approved (recorded as the project rule "main always promotable") are the basis for every guardrail below:

1. Partial or unfinished behaviour sits behind a flag that defaults OFF, gated on the server and on the web: endpoints, consumers, jobs, startup tasks. Never change existing production behaviour unconditionally while the replacement is unfinished.
2. Persisted data changes are additive and rollback-safe: the old build must still read data the new build writes. Use expand/contract; no destructive or renamed fields in one step.
3. No blocking work at startup on large collections (index builds, backfills). Use a migration step or a background build.
4. Production config parity: when a change alters where a flag's value comes from or its default, check the effective production value stays the same.
5. Message contracts stay additive and compatible across a rolling deploy and with separately deployed Lambdas and agents.
6. Treat "would this break a promotion tomorrow?" as a review question with the same weight as correctness.

2. Phase 0: the catch-up promotion (context only)

Main is not promotable today. Making it promotable is a separate track ("Part 1"), and its items are being decided with Chris: the production flag values (A1), deletion (A2), rollback compatibility for SessionTally (A3), pre-creating the startup indexes (A4), the Lambda and bucket versioning (A5), the duplicate env var (A6), plus the deployment strategy, log format, owner-run data checks and user announcements. This plan does not re-plan any of that.

The guardrails do not wait for Part 1. They are designed as ratchets: each one records today's known divergence from production as named "known debt", fails only on new divergence a PR introduces, and reports the remaining debt so Part 1 can burn it down. When the debt reaches zero, a guardrail switches from ratchet mode to strict mode (§4.0).

Part 1 also changes the baseline. Once the catch-up promotion lands, production runs main's runtime-flag code, so the live production flag snapshot (GET https://api.syrf.org.uk/api/runtime-feature-flags) becomes available as a second source for the baseline (§4.2), while production stays in the legacy policy mode. Once production runtime policy is enabled, the live snapshot reflects runtime decisions and cannot prove the baseline; baseline provenance comes from the compatibility report instead (§4.10). Until the catch-up promotion the baseline is computed by rendering production's current chart tags with production's values.

3. Phase 1 overview: the guardrails

ID	Guardrail	What it stops	Runs in	Blocks
G1	Repo rule, CLAUDE.md line, PR template "Promotion safety" section, claude-review question, promotion labels	Partial work leaking; unannounced user-visible changes (A2, A8, A13)	Every PR (humans, agents, claude-review)	Review verdict
G2	Production flag parity + production render diff	Effective production flags or rendered production manifests changing silently (A1, A6, A7, A9, A10)	pr-tests.yml test-charts lane (Juniper)	PR check
G3	Server-gate inventory	New endpoints, hub methods, consumers, jobs, hosted services or startup tasks that are live with production-default flags (A13)	test-dotnet lane (Juniper)	PR check
G4	Rollback compatibility (N-1 read)	Data written by main that the production build cannot read (A3)	test-dotnet (G4a); a cached old-build reader lane (G4b)	PR check
G5	No blocking startup work + startup probe	Index builds and backfills on existing collections at startup (A4)	test-dotnet + test-charts	PR check
G6	Message and contract compatibility	Breaking MassTransit, hub, REST or Lambda/pdf-agent contracts across version skew (A5, A11)	test-dotnet, s3-notifier tests	PR check
G7	Rolling promotion dry-run report	Drift that PR checks cannot see (cluster-gitops changes, accumulated debt)	Scheduled weekly workflow (Juniper)	Report only
G8	Staging mirrors production flags	Staging testing a different configuration from production (A12)	G7's workflow, plus G2's calculator	Report only
G9	Production-flag E2E profile (proposed addition)	Unconditional behaviour changes on core journeys (A2, A8, A13)	e2e-tests.yml on main, nightly	Report, then PR label

3.1 Where the checks run

• Every new check reuses an existing lane where it can, so no new runner route is needed. G2 and the chart half of G5 are steps in pr-tests.yml test-charts (pr-tests.yml:808), which already runs on juniper-ci with [self-hosted, linux, x64, juniper-normal-ci], already verifies Helm (v3.21.3 from camaradesuk/server-config#156), and already has the hosted test-charts-fork counterpart for fork heads. The same steps go into both jobs.
• G3, G4a, the code half of G5 and G6 are ordinary tests in existing .NET test projects, so they run in test-dotnet (Juniper) with no workflow change.
• The plan adds four new self-hosted jobs. All of them follow the existing precedent, statistics-staging-preflight.yml: juniper-ci with the ordinary label set, least-privilege permissions, and short-lived tokens minted per run.

New job	Guardrail	Trigger
promotion-compatibility-report.yml	Trusted collection of the target environment's compatibility report for G2 (§4.2); replaces the vendored-values refresh	Same-repository PR heads that touch G2 inputs, main after a production promotion, workflow_dispatch
promotion-dry-run.yml	G7 and G8 (§4.7, §4.8)	Main only: weekly and workflow_dispatch
rollback-reader job in pr-tests.yml	G4b (§4.4)	Same-repository PR heads on persisted-model paths; weekly through G7
production-smoke.yml	Phase 2 smoke test (§5)	status events from the default branch, plus workflow_dispatch

Each one needs, in the PR that adds it: - a bullet in juniper-runner-routing.md; - an entry in EXPECTED_GLOBAL_SELF_HOSTED_ROUTES (.github/scripts/validate-workflows.sh:937); - one run of the new lane, linked from the PR. - No new runner groups or labels. - Any change to pr-tests.yml must extend the matching routing contract scripts (test-pr-validation-gates-runner-routing.sh) and keep the pinned timeouts exact.

3.2 Bypass and audit trail (shared by every guardrail)

A guardrail can be bypassed only by a committed allow-list entry, never by a label or a re-run. All entries live in one directory so they can be owned and reviewed together:

.github/promotion/
  allowlist.yaml                    # every bypass, one section per guardrail
  known-debt.yaml                   # divergences that exist on main today (Part 1 burns these down)
  staging-flag-overrides.yaml       # staging-only flag differences (G8)
  fixtures/                         # synthetic production-like policy and settings profiles for untrusted lanes

No production values, effective-flag sets or production renders are committed (D1, refined 2026-10-15). They come from the compatibility report fetched by a trusted job and are kept as access-controlled run artifacts.

Each allowlist.yaml entry must carry: the item (flag, render path, index, contract, surface), the change (from → to), a reason, the PR, the approver, and an expiry date. Every check fails on an entry that is missing a field or has expired, so bypasses cannot pile up silently. known-debt.yaml entries name the Part 1 item that removes them.

Ownership matters because agents may self-approve PRs (self-approve.yml). An allow-list entry that an agent adds and approves itself is not a control. syrf has no effective CODEOWNERS today. The only file is src/services/web/.github/CODEOWNERS, and GitHub ignores it, because it reads CODEOWNERS only from the repository root, .github/ or docs/. The "Main Protection" ruleset requires one approval with require_code_owner_review: false.

Recommendation (D7): - add a root .github/CODEOWNERS that owns only /.github/promotion/, naming Chris; - turn on require_code_owner_review in the Main Protection ruleset. Only owned paths are affected, so every other PR keeps today's single-approval rule; - make self-approve.yml refuse PRs that touch .github/promotion/; - delete or move the ignored web CODEOWNERS file, so nobody mistakes it for a control.

4. Phase 1: guardrail designs

4.0 Ratchet mode and strict mode

Each check computes two things:

• PR delta: the result at the PR's base commit versus its head commit. A difference that is not allow-listed fails the PR. This is what makes the checks useful before Part 1 finishes.
• Main versus production: the result at head versus the production baseline. Differences that are in known-debt.yaml are reported in the job summary, not failed. A difference that is neither known debt nor allow-listed fails.

When known-debt.yaml has no entries for a guardrail, that guardrail is in strict mode: head must equal the production baseline plus allow-listed changes.

4.1 G1: repo rule, PR template and review question

What it is.

• A new .claude/rules/promotability.md (frontmatter paths: covering src/**, .github/workflows/** and src/charts/**), about 30 lines: the six rules, the guardrail list with one line each, and how to add an allow-list entry. It must pass python3 docs/scripts/validate-agent-rules.py.
• One CLAUDE.md line. CLAUDE.md is at 199 lines and validate-agent-rules.py requires fewer than 200, so the line extends the existing "Every new feature needs an explicit flag decision" bullet instead of adding a bullet: "…and main must stay promotable to production at every commit; see .claude/rules/promotability.md."
• A "Promotion safety" section in .github/pull_request_template.md:

## Promotion safety
<!-- Would this PR break a production promotion tomorrow? Tick each, or explain. -->
- [ ] Unfinished behaviour is behind a default-off flag, gated on the server and the web (or N/A)
- [ ] Existing production behaviour is unchanged with production flag values (or listed below)
- [ ] Persisted data changes are additive; the production build can read what this writes
- [ ] No new startup work on existing collections (index builds, backfills)
- [ ] Effective production flags and rendered production config are unchanged (or allow-listed)
- [ ] Message, hub, REST and Lambda/agent contracts stay compatible with the deployed versions
- User-visible production change without a flag: <!-- none, or what and why -->
- Production action needed at promotion: <!-- none, or index pre-create / data check / announcement -->

• Two labels: promotion:user-visible (a deliberate, unflagged user-visible change such as A8) and promotion:action (something must happen at promotion time: an index pre-create, an owner-run data check, an announcement). G7 lists every merged PR with either label since the production SHA, so the promotion checklist is built from them rather than rediscovered.
• The claude-review prompt is in-repo (.github/workflows/claude-code-review.yml, the prompt: block, around lines 98–259). Add a priority item after "Data integrity": "Promotion safety: would this break a production promotion tomorrow?" with the six rules in one line each, and add "a change that breaks rule 1–5 with no flag, allow-list entry or recorded decision" to the blocking list. Two constraints: the review runs default-branch workflow code, so the change takes effect only after it merges; and workflow files are ruleset-protected, so the PR needs Chris's merge. If .github/scripts/test-claude-review-comment.cjs or validate-workflows.sh pins prompt text, update those contracts in the same PR.

Fails when. The rule file or CLAUDE.md budget fails validate-agent-rules.py. The review verdict is request changes when a promotion-breaking change has no flag or recorded decision.

Bypass. The PR description records the decision (for example "unflagged by design: security fix, announced at next promotion"), and the PR carries promotion:user-visible.

#	Acceptance criterion	Verification
G1.1	.claude/rules/promotability.md exists with non-empty paths: and states the six rules	python3 docs/scripts/validate-agent-rules.py passes; file review
G1.2	CLAUDE.md stays under 200 lines and points to the rule	wc -l CLAUDE.md < 200; the validator passes
G1.3	The PR template has the "Promotion safety" section above	Open a new PR; the section appears
G1.4	The claude-review prompt asks the promotion question and treats an unflagged rule 1–5 break as blocking	Review a fixture PR that changes an existing endpoint's behaviour unconditionally: the verdict is request changes with a promotion finding
G1.5	Both labels exist and G7 lists labelled PRs since the production SHA	gh label list; G7 report section

4.2 G2: production flag parity and production render diff

What it checks. Render main's charts with production's values in the exact ApplicationSet order, compute the effective flag set every consumer gets, and compare it with the production baseline.

The value-file order comes from cluster-gitops argocd/applicationsets/syrf.yaml:97-103, not from the hierarchy list in gitops-architecture.md, which omits the two config.yaml files (a doc fix to do in the G2 PR):

1. syrf/global.values.yaml
2. syrf/services/<svc>/config.yaml
3. syrf/environments/production/<svc>/config.yaml
4. syrf/services/<svc>/values.yaml
5. syrf/environments/production/production.values.yaml
6. syrf/environments/production/<svc>/values.yaml
   (+ ApplicationSet parameters: image.tag from the service config)

How effective flags are computed (.github/scripts/promotion/effective_flags.py):

1. For API, PM, web and quartz (Identity and pdf-agent are not deployed in production; they join when their production directories exist), run helm dependency build and helm template with the six files above and fixed placeholders for image tags and gitVersion, so renders are deterministic.
2. Read every flag env var from the rendered containers. The flag names come from src/charts/syrf-common/env-mapping.yaml (featureFlag: true entries and featureFlags.* value paths), so new flags are picked up automatically.
3. Server value per service = the rendered env value, or the C# default from FeatureFlags.cs when the env var is absent (SignalRActive = true, all others false).
4. Browser value per flag: if the flag is in the API runtime catalog, the API's value (the API snapshot wins at browser bootstrap); otherwise the web's own rendered value. While production runs the legacy policy mode, production ignores runtime overrides (RuntimeFeatureFlagStore allows mutation and honours overrides only outside production), so the rendered API value is the browser value. Once production runtime policy is enabled (ADR-025) this step computes the accepted baseline only; a legitimate runtime difference is not drift, and live behaviour is checked by the runtime-conformance lane (§4.10). Catalogue dependencies (for example bulkPdfUpload forced off without the notifier authority) are applied.
5. The catalog and its dependency rules are C# (RuntimeFeatureFlagCatalog.cs). To keep the chart lane free of a .NET build, a new API unit test exports them to a committed golden, src/services/api/SyRF.API.Endpoint.Tests/Golden/runtime-feature-flag-catalog.json, and fails when the export is stale. The script reads that file.
6. The browser precedence rule is an input too, not hard-coded in the script. A1 came from a web code change: be28f8c934 (#3030) changed applyRuntimeFeatureFlagSnapshot from "web deployed value, plus overrides" to "the API snapshot value for every catalogued key". No chart changed. A render-only check cannot see that kind of change, so the rule is declared in a committed file, src/services/web/src/app/core/services/runtime-feature-flag-precedence.json, with the mode deployed-plus-overrides or api-snapshot-authoritative. A web spec next to runtime-feature-flags.service.spec.ts drives applyRuntimeFeatureFlagSnapshot and fails when its behaviour does not match the declared mode. The script reads the mode at each ref, so a precedence change shows up as flag changes at the PR that makes it.

The result is a table: {consumer → {flag → value}}, where consumer is browser, api, pm or quartz.

What fails it (G2a, flag parity).

• A flag whose effective value changes for any consumer between the PR's base and head, in either direction, without an allow-list entry.
• A new flag whose effective production value is ON (a new default-on behaviour reaching production) without an allow-list entry. New flags that default OFF pass.
• A flag that is ON in the production baseline and disappears from the render (retiring a flag makes its behaviour permanent: that needs an entry, as newStageSettings would have).
• In strict mode, any difference between head and the production effective flags computed from the current production compatibility report that is not allow-listed.

Three cheap sub-checks in the same step.

• G2b, production render diff. The trusted lane renders base and head with the settings in the target's compatibility report and diffs the normalised renders (image tags, gitVersion values and checksum annotations dropped). The diff is a run artifact and is summarised in the job summary, so the production effect of any chart change is reviewable without committing production renders. Changes to sensitive paths also need an allow-list entry: spec.strategy, probes, replicas, resources, container ports, PodMonitoring, secret references, and removed env vars. A7 (Recreate), A9 (JSON logs, Prometheus) and new env vars would all have shown up here.
• G2c, no duplicate env names. Fail if any rendered container has the same env name twice (A6).
• G2d, every flag is controllable. For each Boolean flag value path, render once with true and once with false; the env value must change. This catches templates such as {{ .Values.featureFlags.apm | default "true" }} that swallow false (A10). Intentionally pinned values need an allow-list entry.

Where it runs. A new step in test-charts and test-charts-fork (pr-tests.yml). Chart change detection (.github/scripts/has-pr-chart-changes.sh) must also select the lane when these change: FeatureFlags.cs, RuntimeFeatureFlagCatalog.cs, the catalog golden, the runtime flag service in the web (runtime-feature-flags.service.ts) and .github/promotion/**. The detection contract .github/scripts/test-pr-chart-change-detection.sh gets fixtures for each.

Inputs (D1, refined by Chris on 2026-10-15). G2 no longer reads a vendored copy. A trusted collection job (promotion-compatibility-report.yml, same-repository heads and main only) authenticates to the target environment's environment-state and feature-policy service with a read-only, environment-bound scope, confirms the environment identity, and fetches a versioned compatibility report: desired cluster-gitops revision and rendered-settings provenance, accepted baseline and catalog identity, policy identity and redacted structure, applied versions and readiness, each with its revision and freshness. It never runs candidate code; the candidate-evaluating G2 steps receive the sanitised report as an artifact. An incomplete, unstable or expired report fails the step (C2, fail closed). Fork heads run G2c, G2d and the fixture-based checks only, with no credential. Design, report contract and failure meanings: runtime feature releases plan §14. Authentication differs by environment today: staging has SyRF Identity with a single-scope machine-client precedent; production is on Auth0 with no machine client the API accepts (plan §14.4, OD-21).

Interim input until the production report exists (D16, a decision for Chris). The production report needs the report endpoint (runtime feature releases plan, Increment B) and a production automation identity (OD-21, Increment C). G2 must not wait for both, or the guardrail for A1 waits with it. Recommended interim: the same trusted collection job (main and same-repository heads only, never fork heads, never running candidate code) reads cluster-gitops main at one resolved SHA with a read-only token minted per run, copies nothing into the repository, and renders production's current chart tags with those values in the exact ApplicationSet order. It passes the effective flag set, the normalised renders and the resolved SHA to the G2 steps as an access-controlled run artifact, scanned by gitleaks, that expires with the run's artifact retention. Nothing is committed and nothing is maintained between runs, so this is a temporary run artifact, not the repository mirror that D1's refinement rejected. G2 switches to the authenticated report per environment as soon as that report exists, the same "until the report exists" fallback that G7 and G8 use. The interim input carries the same fail-closed rule: an unreadable cluster-gitops revision or a failed render fails the step.

Secrets never enter the report: secret-sourced settings appear as "from secret" with a digest. The gitleaks scan (a pinned, checksum-verified binary in RUNNER_TEMP) still runs over every retained report artifact and over .github/promotion/**; any finding fails the job.

Report freshness (replaces the refresh workflow, D2). Each run fetches its own report; there is no committed baseline to refresh. The deployment preflight fetches a fresh report again immediately before a promotion and revalidates it (plan §14.6). While production runs the legacy policy mode, the collection job also compares the anonymous production snapshot with the rendered values and reports a mismatch, which is what catches a mechanism change like A1 on the live side. After production runtime cutover this comparison is replaced by the baseline-provenance and runtime-conformance lanes (§4.10).

#	Acceptance criterion	Verification
G2.1	effective_flags.py reproduces the audit's production table: rendering main at 19c5797af with production values gives the seven A1 flags OFF for the browser, apm/errorTracking/signalRActive ON, PM BulkStudyUpdateRequireCurrentExecutionVersion ON	Run against that commit; compare with the audit table
G2.2	Replay A1: compute G2a with be28f8c934^ as the base and be28f8c934 (#3030) as the head, with the precedence file set to the mode each commit actually implements. It fails and names every catalogued flag that production turns on only in its web values (the A1 flags catalogued at that commit)	Fixture run; red output in the PR body
G2.2b	The precedence spec fails when applyRuntimeFeatureFlagSnapshot's behaviour and the declared mode disagree	Negative fixture in the web spec
G2.3	A new flag that defaults OFF passes; a new flag that defaults ON fails until allow-listed	Contract script fixtures
G2.4	G2b publishes the normalised base-versus-head production render diff as a run artifact and a job-summary section, and fails a sensitive-path change (including spec.strategy) without an allow-list entry; replaying #2976 fails	Contract script fixture + replay
G2.5	G2c fails on the A6 render (main@19c5797af with production values). It passes once #4155 (explicit env deduplicated over generated flags) merges, or Part 1 moves the value into featureFlags. Until then A6 is a known-debt entry noting "cleared by #4155 once merged"	Run at 19c5797af and at #4155's head
G2.6	G2d fails on the apm template at 19c5797af and passes on #4155's head, which honours apm: false. Until #4155 merges, A10 is a known-debt entry noting "cleared by #4155 once merged"	Run at both commits
G2.7	An expired or incomplete allow-list entry fails the step	Fixture
G2.8	Today's main-versus-production differences are seeded in known-debt.yaml and appear in the job summary, not as failures	Job summary on the first PR
G2.9	The step runs in both test-charts and test-charts-fork; the routing contracts pass; timeouts unchanged	validate-workflows.sh; the routing scripts
G2.10a	Interim (D16): the collection job resolves one cluster-gitops SHA, renders production's chart tags, records the SHA in the job summary and passes the effective flags and renders as an artifact; no file under .github/promotion/ holds production values; a fork head gets no token	One workflow_dispatch run; workflow contract fixture
G2.10	Once the report exists: the collection job fetches a Complete report for the target, records its report ID and revision vector in the job summary, and passes it to the G2 steps as an artifact; an Incomplete, Unstable or expired report fails the lane	One workflow_dispatch run plus fault-injection fixtures
G2.11	A planted fake secret in a report artifact fails the scan, and so does one in .github/promotion/**	Fault-injection fixture
G2.12	The collection route is in EXPECTED_GLOBAL_SELF_HOSTED_ROUTES and the routing doc; no job that runs candidate code holds the service credential; fork lanes hold none	validate-workflows.sh passes; workflow contract fixture

4.3 G3: server-gate inventory

What it checks. Everything that is live in a host built with production-default flags, and whether each item is gated or deliberately ungated. Rule 1 says a partial feature's endpoints, consumers, jobs and startup tasks must be gated on the server, not only hidden in the web; the audit found many web-only gates whose endpoints answer anyway (A13).

The test cannot know which code belongs to an unfinished feature, so it works as an inventory golden, the same pattern as StudyWriteLockArchitectureTests (src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests/StudyWriteLockArchitectureTests.cs): every live item must either be gated or appear in an allow-list with a reason.

One test per host (API, PM, quartz) builds the host with production-like configuration and every flag at its default (the API tests already build a host through WebApplicationFactory, for example BffAuthIntegrationTests) and lists:

Kind	How it is listed	Counts as gated when
Controller actions	EndpointDataSource endpoints	[FeatureGated(...)] (Authorization/FeatureGatedAttribute.cs) or a class-level gate such as BffEnabledGate
Hub methods	Reflection over hub classes	A gate attribute, or the existing …Available check pattern, declared with a marker
Consumers	Registered consumer types in the service collection	Not registered while the flag is off, or marked [GatedBy(...)] with a named behavioural test proving it is inert when off
Hosted services	IHostedService registrations	Same as consumers
Quartz jobs	Job types and schedules registered by the quartz host	Same as consumers
Startup tasks	IRunAtInit implementations (TaskRegistry)	Listed separately; G5 owns what they may do

The golden, production-default-surface.golden.txt, has one line per item: kind, identifier, gate (flag name) or ungated: <reason>. The test fails when:

• a new item appears that has no gate and no allow-list reason;
• an item marked [GatedBy] names a test that does not exist (checked by reflection, as the lock test checks its allow-list);
• the golden is stale.

Two smaller changes make this practical. FeatureGatedAttribute.Flag is a closed enum of four flags; it should take any catalogued flag (or gain the missing ones) so gating an action is one attribute. And the seed allow-list records today's ungated-by-design items with their reasons (security hardening, read-only, authenticated per-user state such as ReviewPreferencesController), plus the audit's gaps as known debt.

Web half (D12). Angular lazy routes behind unfinished features should use a flag canMatch guard. A spec that scans *.routes.ts for routes without a guard or allow-list entry is the same pattern; recommendation: a follow-up after the server half.

#	Acceptance criterion	Verification
G3.1	The inventory test lists controller actions, hub methods, consumers, hosted services, Quartz jobs and IRunAtInit tasks for API, PM and quartz	Golden file review
G3.2	A fixture controller action with no gate and no allow-list entry fails the test	Negative fixture in the test project
G3.3	A consumer registered only when its flag is on does not appear in the production-default golden	Unit test with the flag off and on
G3.4	A [GatedBy] naming a missing test fails	Negative fixture
G3.5	FeatureGated accepts every catalogued flag	Unit test over the catalog
G3.6	The seed allow-list covers today's surface; each ungated entry has a reason, and the audit's c-gaps are in known-debt.yaml	Review against the audit's §2 gap list

4.4 G4: rollback compatibility (N-1 read)

What it checks. Documents written by main can be read by the production build's class maps, so production can roll back to the previous release and old and new pods can overlap.

Three options considered.

Option	How it works	Catches	Cost and risk
(a) Load the previous release's assemblies in a test	Build the production tag's data assemblies and load them, with their own MongoDB.Bson, into a separate AssemblyLoadContext; deserialise main's documents with them	Everything the old code cannot read	Two copies of MongoDB.Bson's static class-map registry in one process; registration entry points differ between versions; failures are hard to read. Same build cost as (b)
(b) Stored-document corpus checked against both builds	Main's tests write a synthetic golden corpus (Extended JSON) of every persisted aggregate with every optional field, subtype and enum value populated. A separate reader harness is compiled against the production tag's source and reads the corpus	Added fields, removed or renamed fields, type changes, new enum values and new discriminators	Builds part of the old tree (only the data and core projects); needs a per-baseline adapter for class-map registration. Cacheable by baseline SHA
© Global extra-elements tolerance + coverage test	Every persisted class map ignores or captures unknown elements; a test proves it for every type reachable from a persisted root	Added fields only, and only for rollback to builds that already contain the convention	Cheap. Does not protect rollback to 104eca9, which lacks it

Recommendation: © now as G4a, (b) next as G4b; reject (a). © closes the commonest failure (added fields, A3) cheaply from the first production build that contains it, as a backstop only (see ADR-023): every shape change still ships a migration with a Down, and G4b is the enforcement gate. (b) is the only option that tests the real old code and catches enum and discriminator additions. (a) costs as much as (b) to build and is more fragile, because MongoDB.Bson's registries are process-wide statics.

G4a, tolerance convention and coverage. Tolerance means capture-and-count, never silent drop: types reachable from governed collections capture unknown elements so the reader can count them (ADR-023 backstop section; pre-deploy plan T2, T2a).

• Register tolerance for every SyRF persisted type, either through a global convention pack in src/libs/mongo/SyRF.Mongo.Common/MongoUtils.cs or per class map. There is a real trade-off to record: IgnoreExtraElements drops unknown fields when the old build replaces a whole document, whereas [BsonExtraElements] (what Entity uses) captures them and writes them back. Capture is safer, and it is what lets the reader count reliance, so types reachable from a governed collection, nested value objects included, must capture (or use an equivalent counting convention); silent ignore is allowed only elsewhere. Any field added in an "expand" step must be recomputable or not relied on until the "contract" step, so losing it during a rollback window is harmless. That rule goes into the expand/contract standard (Phase 3).
• A coverage test walks every class map reachable from the persisted aggregate roots (after main's registration runs) and asserts each one ignores or captures extra elements, and that types reachable from a governed collection capture. Exceptions need an allow-list entry.
• A persisted enum and discriminator golden: the member names of every enum stored as a string, and every registered discriminator. Adding a member or discriminator fails until an entry records the expand plan ("ship the reader first, write the new value one release later"). Old code throws on an unknown enum string or discriminator, and © alone cannot catch that.

G4b, old-build reader.

• src/testing/RollbackReader/ (new): a small console project whose project references point at an MSBuild property (BaselineSourceRoot), so it compiles against a checkout of the production tag named by the applied production version in the latest production compatibility report (D1 as refined; until that report exists, the chart tag in cluster-gitops). It registers the old class maps through a per-baseline adapter file and deserialises every corpus document. It reports the first failing document and BSON path.
• The corpus is written by main's tests from fixture builders: synthetic values only, no production or participant data. A test fails when the corpus would change and has not been regenerated (the same pattern as the STJ goldens in #4145).
• It runs in a new job in pr-tests.yml on the ordinary Juniper route, only when persisted model or data paths change (src/libs/project-management/**/Model/**, **/Mongo.Data/**, src/libs/kernel/**, src/libs/mongo/**), plus in G7 weekly. It builds only the old tree's data and core projects, niced and with a processor cap, and caches the built reader by baseline SHA. A new route means the routing doc bullet, EXPECTED_GLOBAL_SELF_HOSTED_ROUTES, a fork counterpart or a same-repository gate, and a linked live run.

#	Acceptance criterion	Verification
G4.1	The coverage test fails for a fixture value object with no tolerance, and passes for every real persisted type after G4a. A nested type reachable from a governed collection with silent IgnoreExtraElements also fails	Negative fixtures; test run
G4.2	Replay A3: a SessionTally document with the four #2467 fields deserialises with main's maps after G4a, and the coverage test would have failed on SessionTally before it	Unit test; run the coverage test at the #2467 merge
G4.3	Adding a member to a persisted enum, or a new discriminator, fails without an entry	Negative fixtures
G4.4	G4b builds the reader against the production baseline and reads the corpus; with today's baseline (104eca9) it fails on SessionTally, matching the audit	Lane run, red output linked; the failure is recorded as known debt until Part 1 resolves A3
G4.5	The corpus is synthetic and complete: every persisted root type has at least one fully populated document	Coverage assertion in the corpus test; fixture review
G4.6	The G4b lane's built reader is cached; a second run on the same baseline does not rebuild it	Two runs, timings in the PR body

4.5 G5: no blocking startup work

What it checks. Startup paths (IRunAtInit, IHostedService.StartAsync, Program.cs before the server listens) do not build indexes on existing collections or run backfills, and every .NET deployment has a startup probe.

Code half.

• A startup index manifest golden. A test runs the IRunAtInit index tasks (API and PM CreateIndexesAndMapping, CreateSagaTtlIndex) against an empty Testcontainers MongoDB, already used in test-dotnet, and writes every collection and index (name, keys, options) to startup-indexes.golden.json. The diff of that file between the production baseline tag and main is the list of indexes a promotion will build. G7 reports it.
• Rule: startup may create an index only on a collection that does not exist in the production baseline manifest (a new, empty collection). An index on an existing collection must be either pre-created by a migration step (the PreSync migration Job of ADR-023, which rejects building it in the background after the server is serving), and startup must tolerate the index already existing with the same name and options. A new shared helper, StartupIndexPolicy, encodes the choice. A source scan, in the style of the lock architecture test, forbids Indexes.CreateOne/CreateMany in startup paths outside the helper.
• Backfills: the same scan forbids UpdateMany, BulkWrite and ReplaceOne loops in startup paths, except allow-listed entries (for example the production-gated DatabaseSeeder).

Chart half.

• Generalise the preview startup probe. _deployment-dotnet.tpl:174-184 already renders a startupProbe, but only when preview.enabled && preview.waitForDatabase. It was added because index creation on a freshly seeded preview database can take more than 60 s, which is the same failure A4 describes. Its shape: httpGet on the liveness probe path, initialDelaySeconds: 10, periodSeconds: 10, failureThreshold: 90 (about 15 minutes), timeoutSeconds: 1. The change renders it for every .NET deployment and moves its numbers into a startupProbe: values block. The preview defaults stay exactly as they are, and the template keeps the preview condition only for the preview-specific defaults. Proposed production default: the same shape with failureThreshold: 30 (about 5 minutes), overridable per service. PROPOSAL for Chris. A probe that tolerates a long start does not make a long index build acceptable: G5's code half still forbids the build.
• A helm unittest in each .NET service chart asserts the probe is rendered.
• The probe changes production manifests, so the change appears in the PR's G2b render diff and is reviewed at promotion like any other.
• Follow-up from the audit (finding 22): give /health/live a predicate so liveness does not run every health check.

#	Acceptance criterion	Verification
G5.1	The manifest golden lists every startup index; the diff between 104eca9 and main shows ProjectId_1_WorkloadShareBucket_1 and ProjectId_1_BulkUpdateLock.OperationId_1 on pmStudy and bulk_pdf_upload_sweep_status on pmProject	Run the test at both commits; compare with the audit
G5.2	A fixture IRunAtInit that creates an index on an existing collection outside StartupIndexPolicy fails the scan	Negative fixture
G5.3	Replay A4: the scan and manifest rule fail at the #2991 and #3909 merges	Run at those commits
G5.4	Every .NET deployment renders a startupProbe, and preview deployments render the same values as today's preview-only probe; helm unittest fails if either changes	Chart tests; render diff of a preview values set before and after
G5.5	The known A4 indexes are known debt linked to Part 1's pre-create step, not failures	known-debt.yaml review

4.6 G6: message and contract compatibility

What it checks. Contracts between components that can run different versions at the same time stay compatible: API and PM during a deploy, PM and the separately deployed Lambda, the off-cluster pdf-agent, the production web against a newer API, and SignalR clients.

G6a, MassTransit and hub contracts.

• #4145 (open) adds MassTransitContractGoldenTests: 56 contracts in API.Messages/PM.Messages, plus 24 hub goldens. G6 reuses those goldens rather than adding new ones. G6a depends on #4145 merging.
• A compatibility classifier compares each contract's shape (property names, types, nullability, and the message URN urn:message:<Namespace>:<Type>) between the production baseline tag and head. Adding an optional property passes. Removing or renaming a property, changing a type, or renaming or moving a message type fails, unless an allow-list entry records a two-step plan. This is shape-based, not byte-based, so harmless formatting changes do not fail.
• The Wolverine switch-over in ADR-021 is the one planned non-rolling change: a single coordinated switch with no side-by-side running. It gets an explicit allow-list entry tied to ADR-021's rehearsed runbook, not an exemption from the rule.

G6b, separately deployed components (Lambda, pdf-agent).

• Environment fixtures, not just code fixtures. The s3-notifier tests gain one S3 event fixture per real bucket configuration: versioned (staging), unversioned/suspended (production, versionId null), and each library file type. A5 fails this test on main today, so it starts as known debt linked to Part 1's Lambda fix and becomes required when that lands.
• A cross-deployable enum golden. Shared enums that cross a deployment boundary (for example LibraryFileType in SyRF.SharedKernel, read by the Lambda) are listed with the minimum deployed version of each consumer. A new member fails unless an entry says the consumer is updated first, or the producer gates the value. This catches A11 (RIS).
• A production prerequisites manifest, .github/promotion/production-prerequisites.yaml: what main assumes about production's environment (bucket versioning, IAM permissions such as s3:GetObjectVersion, Mongo roles for createCollection and createIndex, broker permissions). G7 checks each item against cluster-gitops values (for example the s3-notifier chart's versioning value) and marks any it cannot check as UNKNOWN, never as passed.
• The version-skew rules go into .claude/rules/s3-notifier.md and .claude/rules/pdf-agent.md: both components lag the cluster, so messages they publish must be readable by old and new PM, and messages they consume must be producible by both.

G6c, REST contract against the production web. The web deploys after the API and old tabs stay open, so the API must stay compatible with the production web for at least one release. G7 diffs the production tag's swagger.json with main's and lists breaking changes (removed routes or properties, narrowed types). Making it a PR check is a follow-up.

#	Acceptance criterion	Verification
G6.1	The classifier passes an added optional property and fails a removed or renamed property, a type change and a renamed message type	Fixtures over the #4145 golden format
G6.2	The classifier compares head with the production baseline tag, not only with the PR base	Fixture where two PRs each make half of a breaking change
G6.3	The s3-notifier suspended-bucket fixture fails on main today (A5) and is listed as known debt	Test output; known-debt.yaml
G6.4	Replay A11: adding Ris to LibraryFileType fails the cross-deployable enum golden	Run at the #2971 merge
G6.5	The prerequisites manifest exists, and G7 reports bucket versioning for production as false against a code requirement	G7 report
G6.6	The Lambda and pdf-agent rule files state the skew rules	validate-agent-rules.py; review

4.7 G7: rolling promotion dry-run report

What it does. Once a week, render production with main and diff everything a promotion would change against what production runs now, so drift is visible weekly instead of being discovered at promotion time.

Workflow: promotion-dry-run.yml.

• Triggers: schedule (weekly, Monday early morning UTC; PROPOSAL) and workflow_dispatch. Main-only (github.ref == 'refs/heads/main').
• Route: juniper-ci, [self-hosted, linux, x64, juniper-normal-ci], following statistics-staging-preflight.yml. Needs the routing doc bullet, EXPECTED_GLOBAL_SELF_HOSTED_ROUTES, and a linked live run.
• Permissions: contents: read, issues: write (to update one report issue). The production compatibility report through the read-only automation scope (runtime feature releases plan §14.4); until that identity exists, a cluster-gitops read-only token minted per run plus anonymous reads. No production database access. No cluster access in the MVP (D5).
• Concurrency: one run at a time, no cancellation.

What it reports.

Section	Source
Production versus main: SHAs, commit and merge counts, days since the last promotion	Production compatibility report (applied versions), git
Effective flag changes per consumer (G2), with known debt and allow-list status	G2 calculator: production chart tags versus main's charts, both with the settings in a fresh production compatibility report
Rendered manifest diff: strategy, probes, replicas, resources, ports, new or removed env vars, PodMonitoring, secret references	helm template at both chart versions
New startup indexes and the collections they touch	G5 manifest diff between the production tag and main
N-1 read status	G4b run against the current baseline
Contract changes (MassTransit, hub, Lambda enums, REST breaking changes)	G6 classifiers; swagger.json diff
Production prerequisites, each PASS, FAIL or UNKNOWN	G6b manifest against cluster-gitops
Staging versus production flag parity	G8
PRs labelled promotion:user-visible or promotion:action since the production SHA	GitHub search
Live checks: deployed version equals the cluster-gitops desired version; production flag snapshot equals the baseline (after Part 1, legacy policy mode only); after runtime cutover, runtime conformance (§4.10)	The compatibility report's desired-versus-applied sections; anonymous reads until the production automation identity exists

Output. It edits the body of one pinned issue, "Production promotion dry-run", and adds a comment only when the verdict changes, so it does not create noise. The full diffs go to a run artifact. The verdict is GREEN (nothing outside the allow-list), AMBER (only known debt or listed actions), or RED. The job fails only on tooling errors. A section it cannot compute shows UNKNOWN: an empty or failed input never reads as GREEN.

Phase 2 also attaches the latest report to the rolling draft production PR (§5).

#	Acceptance criterion	Verification
G7.1	A workflow_dispatch run on main produces the report with every section above	Run URL; issue body
G7.2	Run against today's state, the report lists A1, A4, A5, A6, A7 and A9	Compare with the audit
G7.3	A section whose input fails shows UNKNOWN and the verdict cannot be GREEN	Fault-injection fixture (for example an unreachable endpoint)
G7.4	The job holds only contents: read and issues: write, plus the read-only report scope (or, until it exists, a read-only cluster-gitops token)	validate-workflows.sh; workflow review
G7.5	The route is in EXPECTED_GLOBAL_SELF_HOSTED_ROUTES and the routing doc	validate-workflows.sh passes
G7.6	Re-running with no change edits nothing and adds no comment	Two consecutive runs

4.8 G8: staging mirrors production flags

What it checks. Staging's effective flag values equal production's, except for listed staging-only overrides, so staging exercises what production users get.

• Staging effective values = the G2 calculation with the settings in staging's compatibility report plus staging's runtime policy from the same report (until the staging report exists, the anonymous GET https://api.staging.syrf.org.uk/api/runtime-feature-flags read the audit used).
• Expected = production's effective values, overlaid with .github/promotion/staging-flag-overrides.yaml. After production runtime cutover, "production's effective values" means the accepted production baseline plus a production-like synthetic profile from .github/promotion/fixtures/; production's live policy is never copied to staging, and staging policy is never overwritten to manufacture parity. Each entry carries the flag, the staging value, a reason (for example "FEAT-024 staging pilot"), an owner and an expiry date.
• Direction matters. A flag ON in production but OFF on staging (A12: newScreeningOverview, newProgressIndicators) means staging is not testing production's behaviour; it can be allow-listed only with an explicit reason. A flag ON on staging but OFF in production is the normal pilot case and needs only a listed entry.
• It runs inside G7's workflow, and daily if Chris wants faster feedback. Report only (D6): a mismatch turns G7 AMBER or RED and is listed with the fix (a cluster-gitops staging values change or a runtime override change). Staging is not mission critical, so fixing it needs no maintenance window.

#	Acceptance criterion	Verification
G8.1	Run against today's staging, G8 reports newScreeningOverview and newProgressIndicators as production-ON, staging-OFF	G7 report
G8.2	Flags that are ON on staging only through Mongo overrides are reported as overrides, separately from deployed values	G7 report
G8.3	An expired staging override entry is reported as a mismatch	Fixture
G8.4	The seed staging-flag-overrides.yaml lists today's intended pilots (FEAT-024, BFF, bulk PDF, keyword highlighting) with owners	File review

4.9 G9: production-flag E2E profile (proposed addition)

G1–G8 cannot mechanically detect an unconditional change to existing behaviour, such as A2 (deletion 503) or A8 (new 403s for ordinary members). The hermetic E2E stack can, if it runs with production's effective flags.

• An E2E profile that loads the production effective flags (from the latest production compatibility report while production is in the legacy policy mode; afterwards the accepted baseline plus synthetic fixtures, so an admin toggle never changes a result) into the stack's configuration and runs the smoke suite's core journeys as an ordinary member and as an admin: sign in, open a project, screen and annotate, export, delete a project and a search, open full statistics.
• It runs nightly on main on the existing E2E listeners (Bramble), never against staging or production. A failure opens or updates an issue and is listed in G7.
• Not part of the MVP: E2E capacity is shared and the full suite is budget-limited. Decision D13.

#	Acceptance criterion	Verification
G9.1	The profile starts the stack with the production effective flags	Stack config dump in the run
G9.2	On main today, the delete journeys fail with 503 (A2)	Nightly run, linked
G9.3	The run uses only the hermetic stack	Workflow review

4.10 Lanes after production runtime cutover (from the runtime feature releases plan)

These lanes come from Chris's 2026-10-15 proposal. They replace the anonymous-snapshot-equals-render check once production runtime policy is enabled, and are designed in the runtime feature releases plan §14.7. Before cutover, the legacy comparison stays where its assumptions hold.

Lane	Input	Failure means
PR policy and contract tests	Synthetic fixtures, no credentials	Wrong rules, permissions, dependencies, safe operation or contracts
Trusted candidate compatibility	Immutable report for the target, from the trusted collection job	The candidate cannot preserve enabled behaviour or understand accepted configuration or policy
Deployment preflight	A fresh report plus revalidation; records the deployment compatibility constraint	Earlier evidence stale, incomplete or incompatible
Baseline provenance	Accepted baseline identity and its reviewed source, from the report	The accepted baseline is not the reviewed one
Runtime conformance and health	Accepted policy versus each host's acknowledged state	Propagation, stale-state or lifecycle failure, never a Git or default mismatch
Staging rehearsal	The same contracts and checks against staging state plus synthetic profiles	Broken check workflow or missing coverage; never authorises production

Runtime toggles never require a repository update and never change deterministic PR fixtures. Production configuration changes invalidate the relevant preflight. Production's current build (104eca9) has no report endpoint, so the catch-up promotion and the first report-capable deploy are validated by the legacy checks (G2 on the D16 interim input, and G4); the report-based lanes take over from the next promotion.

5. Phase 2: regular small promotions

Goal: promote on a fixed cadence with a checklist that is mostly automated, so each promotion is small and boring. This starts after the Part 1 catch-up promotion.

Cadence (D4). Weekly or fortnightly; Chris decides. Recommendation: fortnightly at first, then weekly once two promotions in a row needed no manual fix. A skipped cadence slot is fine; drift is still visible in G7.

What stays the same. The rolling draft PR in cluster-gitops (promote-to-production in ci-cd.yml, branch production-promotion, label requires-review), the human "ready for review" step, the code-owner approval, and the administrator merge. Merging is the deployment. Agents never mark it ready, approve it or merge it.

Prerequisites from other plans. Build and CI plan PR-2 must land first: promotion copies each staging config.yaml wholesale (including staging's deploymentId, F1), and snapshots cluster-gitops before the staging merge completes (F1b). F2 (Lambda releases) and F11 (stale promote-production.yml) matter too.

The promotion checklist, generated into the rolling PR body by promote-to-production:

1. The latest G7 report, with its verdict. RED blocks; AMBER items each need an owner decision.
2. Release notes generated from conventional commits between the production SHAs and the candidate (grouped feat/fix, with PR links).
3. Every promotion:user-visible PR (for the announcement) and every promotion:action PR (the action list).
4. Index pre-creates needed, from the G5 manifest diff, with the runbook.
5. The Lambda code pin decision (lambda.code.sha256 stays production's unless deliberately changed).
6. Deploy order: quartz, then PM and API back to back, then web (API before web, never web first), then the s3-notifier chart; Lambda code separately.

Post-deploy smoke test. A new workflow, production-smoke.yml.

What signals a production deploy. The ArgoCD PostSync hook (src/charts/syrf-common/templates/_postsync-notify.tpl, included by the API, PM, web and quartz charts) posts a commit status on the syrf commit in that service's deploymentNotification.commitSha. The status has context argocd/deploy-production, state success and description Deployed <service>:<version> to production (production-promotion-and-notifications.md, "Deployment Success Notifications"). The hook also posts a GitHub deployment status, but only when deploymentId is set, and production's deploymentId is the staging Deployment that promotion copies across (build-and-CI plan F1). deployment_status is therefore not a production signal and is not used here.

Trigger. on: status, plus workflow_dispatch. The job runs only when github.event.context == 'argocd/deploy-production' && github.event.state == 'success'. Two GitHub rules shape this: - status workflows always run the workflow file from the default branch, so the smoke job can only be changed by merging to main, and a branch cannot test it through this trigger. Its first live run is a workflow_dispatch on main. - Statuses created with GITHUB_TOKEN do not start workflows. The hook authenticates as the GitHub App, so its statuses do. The job must never post its own statuses with GITHUB_TOKEN and expect a chain.

Waiting for all four services. The hook fires once per service, possibly on different commit SHAs (each service's config.yaml has its own commitSha), so one event is not "the deploy finished". Each event runs a cheap gate step first: 1. Read the production config.yaml for API, PM, web and quartz from cluster-gitops main with a read-only token: the expected imageTag and commitSha for each. 2. For each service, list the statuses on its commitSha (GET /repos/camaradesuk/syrf/commits/<sha>/statuses, the full list rather than the combined status, because all four services share one context and the combined view keeps only the latest per context). Look for an argocd/deploy-production success whose description names that service and its expected version. 3. If any service is missing, exit successfully as "waiting". A later event will complete the set. 4. When all four match, run the smoke checks. A concurrency group (production-smoke, no cancellation) plus an idempotency marker stop duplicate runs when two events complete the set together. The marker is a job summary entry, and a run artifact keyed by the four service:version pairs; the gate skips when the artifact already exists. 5. A deploy that never completes the set is caught by G7: it compares desired and deployed versions and reports a production deploy that has been partial for more than 30 minutes.

Alternatives considered: - Create a real GitHub Deployment from cluster-gitops. When the production PR merges, a cluster-gitops workflow creates a syrf Deployment per promotion, and the hook reports against it. That gives one deployment_status event per promotion and fixes F1 properly, but it needs a cluster-gitops workflow, an App permission and a change to how deploymentId reaches production. It is a good Phase 3 upgrade. - repository_dispatch from the hook or from cluster-gitops. An explicit event with a payload such as "all services synced". It is simple to consume, but the sender needs a token with contents: write on syrf, and something still has to decide when "all" have synced.

Checks. Read-only and anonymous in the MVP (D8): - every service's /health/live and /health/ready; - the deployed version equals the expected tag; - the production flag snapshot equals the effective flags computed from the latest production report (legacy policy mode only; after runtime cutover, the runtime-conformance lane in §4.10); - the web index.html and config load; - the SignalR negotiate route answers 401 when unauthenticated.

No writes and no real accounts.

Watch window. For 60 minutes after the merge: the Sentry release's crash-free rate and new issues, the 5xx ratio and API p95 latency (Grafana, once Prometheus is live after Part 1). Manual in Phase 2; automated in Phase 3.

Documented rollback (a new section in production-promotion-and-notifications.md):

1. Revert the production promotion PR in cluster-gitops (an administrator merges the revert). ArgoCD syncs the previous image and chart tags.
2. This is safe only when G4b is green for the previous release; if G4b is red, the promotion is blocked until a repair or compatibility image exists (that is what G4 is for).
3. Lambda: restore the previous lambda.code.sha256 and sha256Version pair.
4. Data: none needed while changes follow expand/contract.
5. Rehearse the revert on staging once (staging is not mission critical, so this needs no window).

#	Acceptance criterion	Verification
P2.1	Chris has chosen a cadence, and two consecutive promotions follow it	cluster-gitops merge history
P2.2	The rolling PR body contains the G7 verdict, release notes, the labelled-PR lists and the index actions	Rolling PR body
P2.3	production-smoke.yml runs once, automatically, after all four services report argocd/deploy-production success for the expected versions; earlier events exit as "waiting"; it makes no write calls	Run URL; code review of the calls
P2.4	The rollback steps are documented, and a staging revert has been rehearsed with timings	Doc PR; rehearsal run notes
P2.5	Lead time from a commit landing on main to production is under two cadence periods	G7 "days since promotion"
P2.6	No production promotion needed a fix discovered during the promotion itself (two in a row)	Promotion notes
P2.7	Production merges stayed with the administrator; no automation marked the PR ready, approved or merged it	cluster-gitops PR timeline

6. Phase 3: continuous delivery

Goal: every commit on main is releasable, and production deployment is one approved action. The approval stays: continuous delivery, not continuous deployment.

1. Every main commit is releasable. G1–G8 run in strict mode (no known debt) and G7 is GREEN on main continuously.
2. One approved action. Chris marks the rolling PR ready and an administrator merges it. Everything before (checklist, notes, pre-creates) and after (smoke, watch, rollback criteria) is automated.
3. Automated production smoke test. As Phase 2, plus authenticated read-only checks if Chris approves a dedicated read-only synthetic account (D8). Never writes, never real accounts.
4. Automated rollback criteria. Sentry release health (crash-free sessions dropping by more than an agreed margin, a new issue above an agreed event rate) and Grafana (5xx ratio, p95 latency) within the watch window. When a criterion trips, automation opens a revert PR in cluster-gitops and alerts Chris. Merging it stays a human action unless Chris decides otherwise (D9). Thresholds are a PROPOSAL set after a few weeks of baseline data.
5. Rolling updates. Switch production API and PM from Recreate to RollingUpdate once old and new builds can coexist: G4b and G6 green for every promotion over an agreed period (D10). Until then Recreate's outage is the price of skipping coexistence.
6. Expand/contract as the standard (a new how-to): add fields and readers first; write new shapes one release later; remove old shapes only after production has run the reader for a full release. Fields added in the expand step are recomputable or unused until contract (the G4a trade-off).
7. Migration jobs instead of startup work. Index builds and backfills run as a pre-deploy Job using a migrations entry point, idempotent and resumable, with explicit Down migrations for rollback. G5's StartupIndexPolicy points at it. Designed in ADR-023 (an ArgoCD PreSync Job in the API and PM charts, using the service image) and planned in the pre-deploy migrations plan, which brings this item forward so the catch-up promotion can use it.
8. Release notes from conventional commits, attached to the rolling PR and published as a GitHub release per promotion (D11). GitVersion already parses the commit types; the build plan's F12 (scope regex) must land first so feat(scope): commits are classified correctly.
9. DORA metrics, computed monthly by a scheduled job into the G7 issue or a dashboard:
10. Deployment frequency: merges of the cluster-gitops production promotion PR.
11. Lead time: from when a commit lands on main to when it is running in production. The deploy time is the created_at of the last of the four argocd/deploy-production success commit statuses for that promotion, the same signal as the smoke test. Not GitHub deployment statuses, which point at a copied staging Deployment (build-and-CI plan F1). The production PR's merge time is the fallback when a status is missing; the report says which source each figure used.
12. Change failure rate: promotions followed by a revert or a hotfix within 7 days.
13. Time to restore: from the trip or report to the restoring deploy, timed the same way.

Targets are a PROPOSAL: weekly deploys, lead time under 7 days, change failure rate under 15 %, time to restore under 1 hour. If the real-Deployment alternative in §5 is adopted, lead time can move to deployment statuses.

#	Acceptance criterion	Verification
P3.1	known-debt.yaml is empty and every guardrail runs in strict mode	File; check output
P3.2	G7 has been GREEN on main for four consecutive weeks	Issue history
P3.3	A production deploy needed exactly one human approval and merge, with no manual pre- or post-steps	Promotion notes
P3.4	A tripped rollback criterion opens a revert PR within 5 minutes (exercised on staging)	Staging drill run
P3.5	Production API and PM run RollingUpdate and a deploy causes no outage	Uptime check during a deploy
P3.6	No startup path creates an index on an existing collection; migrations run as a PreSync job	G5 scan; chart render
P3.7	DORA metrics are published monthly	Report link
P3.8	Production deploys still require explicit human approval	cluster-gitops ruleset and CODEOWNERS

7. MVP boundary and sequencing

7.1 MVP

The MVP is the smallest set that would have caught the audit's top findings at the PR that introduced them:

1. G1: rule, CLAUDE.md line, PR template, labels, claude-review question.
2. G2: flag parity (G2a), production render diff (G2b), duplicate env (G2c), controllability (G2d), with the trusted collection job on the interim cluster-gitops input (D16) until the authenticated report exists (D1 as refined).
3. G4a: tolerance convention, coverage test, persisted enum and discriminator golden.
4. G5a: startup index manifest, startup scan, startup probe.
5. G6b-lite: the s3-notifier environment fixtures and the cross-deployable enum golden.

MVP acceptance: every criterion in G1.1–G1.4, G2 (with G2.10a in place of G2.10 until the report exists), G4.1–G4.3, G5.1–G5.5, G6.3 and G6.4 passes, and each replay row in the table below fails at its original PR as stated. G1.5 is excluded because its verification (G7 listing labelled PRs) depends on G7, which is outside the MVP. The MVP still creates the two labels. G1.5 is met when PR-5 (G7) lands.

Audit finding	Guardrail that catches it	Would have failed at	Mechanical?
A1 seven web flags switch off	G2a flag parity (and the live cross-check after Part 1, legacy policy mode)	The PR that made the API snapshot authoritative	Yes
A2 deletion 503	G1 (template + review question + promotion:user-visible); G9 later	#2843	Review now; E2E with G9
A3 SessionTally blocks rollback	G4a coverage test; G4b for the real old build	#2467	Yes
A4 startup index builds on pmStudy	G5a manifest + scan; startup probe	#2991, #3909, #2788	Yes
A5 Lambda needs bucket versioning	G6b suspended-bucket fixture; prerequisites manifest in G7	#2841	Yes
A6 duplicate env var	G2c	#3387	Yes. The instance itself is cleared by #4155 once merged; G2c stops a recurrence
A7 Recreate default	G2b sensitive-path rule	#2976	Yes
A8 authorization tightenings	G1 promotion:user-visible label, aggregated by G7; G9 later	#3243 and others	Visibility, not a block (they are intended)
A9 JSON logs and Prometheus go live	G2b render diff	#3515, #3597	Yes
A10 apm cannot be switched off	G2d	The template line	Yes. The instance itself is cleared by #4155 once merged; G2d stops a recurrence
A11 RIS and the old Lambda	G6b cross-deployable enum golden	#2971	Yes
A12 staging differs from production	G8	n/a (configuration)	Yes, report
A13 ungated endpoints and producer writes	G3 inventory (post-MVP); G1 meanwhile	#3579, #3898	Yes with G3

7.2 Sequencing (most protection for least work first)

flowchart LR
  P1[PR-1 G1 rule + template] --> P2[PR-2 G2a parity, interim cluster-gitops input]
  P1 --> P3[PR-3 G4a tolerance + coverage]
  P2 --> P4[PR-4 G2b-d render diff]
  P2 --> P5[PR-5 G7 dry-run report]
  P4 --> P6[PR-6 G5 startup manifest + probe]
  P3 --> P7[PR-7 G6b Lambda fixtures + enum golden]
  S4145{{#4145 merged}} --> P8[PR-8 G6a contract classifier]
  P5 --> P9[PR-9 G8 staging mirror]
  P1 --> P10[PR-10 G3 server-gate inventory]
  P3 --> P11[PR-11 G4b old-build reader]
  P2 --> P12[PR-12 G9 E2E profile]
  P10 --> P12

Order	PR	Guardrail	Effort	Depends on	Why this position
1	PR-1	G1	S	none	Costs almost nothing; changes what every human and agent checks from the next PR
2	PR-2	G2a + trusted collection job with the interim cluster-gitops input (D16)	M	PR-1 and Chris's D16 decision. It does not wait for the runtime feature releases plan: moving G2 to the authenticated report is a later swap of the input, per environment, once the report endpoint (that plan's Increment B) and, for production, the automation identity (Increment C, OD-21) exist. If Chris rejects D16, PR-2 and everything that depends on it (PR-4, PR-5, PR-9, PR-12 and the MVP's A1 guardrail) wait for those increments	Catches the largest finding (A1) and builds the calculator G7, G8 and G9 reuse
3	PR-3	G4a	M	PR-1 only	Independent of the chart work, so it starts straight after PR-1, in parallel with PR-2. Closes the rollback class going forward
4	PR-4	G2b–G2d	S	PR-2	Same lane and script as G2a; catches A6, A7, A9, A10
5	PR-5	G7	M	PR-2	Makes everything visible weekly, including cluster-gitops drift that PR checks cannot see. Not in the MVP, but cheap once PR-2 exists. Sections for later guardrails show UNKNOWN until those land
6	PR-6	G5a + startup probe	M	PR-4	Catches A4. The probe changes every .NET production render, so it needs the G2b render diff in place to review that change; it does not depend on G4a
7	PR-7	G6b-lite	S	PR-3	Catches A5 and A11. Its cross-deployable enum golden reuses PR-3's persisted-enum golden helper
8	PR-8	G6a	M	#4145	Reuses #4145's contract goldens
9	PR-9	G8	S	PR-5	Reuses the G2 calculator inside G7's workflow
10	PR-10	G3	L	PR-1	Technically independent, but placed late because it is the largest: host building and a big seed allow-list
11	PR-11	G4b	L	PR-3	Builds the old tree; worth it once G4a is in
12	PR-12	G9	M	PR-2, PR-10	Needs the production effective flag set (PR-2), G3's inventory (D13) and Chris's decision

After PR-1, PR-2 (given D16) and PR-3 run in parallel. After PR-2, PR-4 and PR-5 run in parallel. PR-6 waits for PR-4, and PR-7 waits for PR-3. PR-8 can start whenever #4145 merges. PR-10 can start at any point if capacity allows.

7.3 Common acceptance criteria (every guardrail PR)

#	Criterion	Verification
C1	The check is written test-first: a fixture or replay fails before the change and passes after	Red and green output in the PR body
C2	Each guardrail fails closed: a missing input, failed render or unreadable file fails the check, never passes it	Fault-injection fixture
C3	.github/scripts/validate-workflows.sh and actionlint -shellcheck= pass for any workflow change; routing contracts updated; a routing change triggers its lane with a linked run	Output and run URL
C4	No new runner groups or labels; existing timeouts stay pinned exactly	Routing contracts
C5	Nothing touches production: no production values edit, no cluster write, no production database access. The rolling production PR is never readied, approved or merged by an agent	Review
C6	Docs updated in the same PR: this plan's status markers, the affected how-to, .claude/rules/promotability.md	validate-docs.sh --skip-indexes
C7	Local commands stay narrow and niced (single test project or single script); no full-solution build on the CI host	Commands listed in the PR body
C8	pr-review-settled.sh exits 0 on the head and Test Summary is green	Gate output
C9	Fixtures and goldens are synthetic: no production or participant data	Review

7.4 Flag decision

Not flagged. The guardrails are tests, CI steps, chart assertions and docs with no runtime feature surface. Three items do change runtime behaviour and carry their own decision in their PRs: the BSON tolerance convention (G4a; reader tolerance only, strictly more permissive), the startup probe (G5; a chart change reviewed through G2b and approved at promotion), and StartupIndexPolicy (G5; changes when indexes are built, covered by the manifest golden). Production rollout of any of them happens only through a Chris-approved promotion.

8. Risks and open decisions

8.1 Risks

Risk	Mitigation
Allow-list fatigue: entries become a rubber stamp	Required reason, approver and expiry; CODEOWNERS on .github/promotion/ (D7); G7 lists entries near expiry
Golden churn makes PRs noisy	Goldens hold shapes only (no production renders are committed; G2b is a run-artifact diff); one script regenerates each; the sensitive-path list for G2b is short
Environment evidence goes stale	No committed baseline; each run fetches its own report with a revision vector and expiry; the deployment preflight refetches and revalidates immediately before promotion
Ratchet mode hides old debt	known-debt.yaml is seeded from the audit and every entry links a Part 1 item; G7 shows the count each week
The checks only see what renders	G7's live checks (deployed version, live flag snapshot) and G9 cover behaviour; the plan does not claim mechanical coverage of unconditional behaviour changes until G9
Juniper capacity	Steps reuse existing lanes; G4b builds only part of the old tree, cached by baseline SHA and run on path changes; G7 is weekly
G4b breaks when old registration code differs	A per-baseline adapter file; when the adapter cannot be written, G4b reports UNKNOWN and G7 shows it
The G6a classifier depends on an open PR	Sequenced after #4145; the rest of the MVP does not depend on it
A guardrail itself becomes a promotion risk (for example the startup probe)	Every chart change goes through G2b and is approved at promotion

8.2 Decisions for Chris

#	Decision	Recommendation
D1	Source of production environment state for deployment checks	Refined by Chris on 2026-10-15: an authenticated environment-state and feature-policy service shared by the application, admin UI and deployment checks through scoped views, with the same contract and check workflow in staging and production and separate state and credentials. Supersedes option A (vendoring production values into syrf). Design: runtime feature releases plan §14; open details are its OD-20 to OD-23
D2	When to refresh the baseline	Superseded by D1's refinement: no committed baseline; each lane fetches a fresh report, and the preflight refetches immediately before promotion
D3	N-1 approach	© tolerance + coverage now, as a backstop only (ADR-023); (b) old-build corpus reader next, as the enforcement gate; reject (a)
D4	Promotion cadence	Fortnightly first, weekly after two clean promotions
D5	Production read access for G7	Anonymous endpoints until the production automation identity exists (runtime feature releases plan OD-21); the workload collector needs a read-only production Kubernetes identity like the staging preflight's (OD-23), a camarades-infrastructure change
D6	Does G8 block anything	Report only; no staging promotion block
D7	Who approves allow-list entries	Chris. Add a root .github/CODEOWNERS owning /.github/promotion/ (the only CODEOWNERS file today, src/services/web/.github/CODEOWNERS, is ignored by GitHub), turn on require_code_owner_review in the Main Protection ruleset (it affects owned paths only), and make self-approve.yml refuse PRs that touch that directory
D8	Production smoke identity	Anonymous checks only; decide later on a read-only synthetic account in the production Auth0 tenant
D9	Automated rollback	Automation opens a revert PR and alerts; a human merges it
D10	When to move production to RollingUpdate	After G4b and G6 are green for every promotion over an agreed period (proposal: four promotions)
D11	Where release notes go	The rolling PR body, plus a GitHub release per promotion. These are engineering release notes; user-facing release communication for runtime-released features (What's New, user guide, announcements) is owned by the runtime feature releases plan §12
D12	Web route guard inventory in the MVP	No; a follow-up after G3's server half
D13	Adopt G9 (production-flag E2E profile)	Yes, nightly on Bramble once G3 lands
D14	Startup probe defaults and G7 schedule	10 s period, 30 failures; weekly Monday early morning UTC
D15	claude-review severity for a promotion break	Blocking, as proposed in G1
D16	G2's input until the authenticated production report exists	A trusted main or same-repository job reads cluster-gitops at one resolved SHA with a per-run read-only token and renders production's chart tags for that run only; results are an access-controlled run artifact; nothing is committed or maintained. G2 moves to the report once it exists. Rejecting this makes the MVP's A1 guardrail wait for the runtime feature releases plan's Increments B and C

9. Overlap with other plans

Plan	Overlap
Build and CI plan	Phase 2 depends on PR-2 (F1, F1b: copy release fields only; wait for the staging merge), F2 (Lambda releases), F11 (stale promote-production.yml) and F12 (GitVersion scope regex, for release notes). Its common criteria C1–C8 are the model for §7.3
System.Text.Json migration plan and #4145	G6a reuses its 56 MassTransit contract goldens and 24 hub goldens. Its stjMvcSerializer and stjHubProtocol switches are exactly the kind of production-default-off flags G2 protects
Realtime server plan and realtime client plan	realtimePipelineV2 is read once at startup and registers change-feed readers and the dispatcher: G3 must show them absent in the production-default inventory. Hub contract changes go through G6a
ADR-021: messaging after MassTransit v8	The Wolverine switch-over is the one planned non-rolling change; G6a allow-lists it against the ADR's rehearsed runbook, including the Lambda and ARRNC agent deploy order
Persistence and messaging plan	Data-integrity changes follow the expand/contract standard (Phase 3) and are checked by G4
ProjectStatistics decision brief	FEAT-024 is the largest dark feature on main; its 25 flags and the empty production allowlist are G2 baseline entries, and its staging pilot is the main G8 staging override
Deletion lifecycle rollout	A2 is a Part 1 decision (D7 there); G9 is the guardrail that would catch the next one
Runtime feature releases plan and ADR-025 (Proposed)	Owns the environment-state and feature-policy service that G2, G7, G8 and G9 read (D1 as refined), production runtime policy (which makes G2 step 4 conditional on the legacy mode), the post-cutover lanes in §4.10, and user-facing release communication (What's New, user guide, announcements) for runtime-released features. It replaces the informal "flag-aware What's New and user guide" idea
Pre-deploy migrations plan and ADR-023	Delivers Phase 3 item 7 early, for A3 and A4: the migration Job, the SessionTally Down, StartupIndexPolicy, and G5's startup probe (default off)

10. Follow-ups found while writing this plan

These are not part of this plan's deliverable:

• gitops-architecture.md lists a four-level value hierarchy that omits the two config.yaml files the ApplicationSet loads (syrf.yaml:97-103). Fix it in the G2 PR, since G2 depends on the real order.
• production-promotion-and-notifications.md's "Future Enhancements" (automated rollback, deployment metrics, progressive delivery) should point to this plan once it is approved.