Skip to content

ADR-023: Pre-deploy database migrations

Status

Approved, 2026-10-14. Chris agreed every recommendation in the plan's decisions table on 2026-10-14 (D1–D9 as recommended; D10 deferred, revisit before activeReviewerTrackingEnabled is turned on anywhere). This ADR answers Chris's decision of 2026-10-14:

"Index builds: I think automating is def preferable but in this case part of the deployment steps should do required db checks/migration in a considered way before deploying app pods that expect certain indexes and schema. These additional db prep steps/jobs should be able to be rolled back. and also be well tested!"

He also prefers explicit migrations with a considered rollback over relying on old builds tolerating new fields: "as long as we ensure migrations is taken care of (and rollback explicitly considered)".

Amended 2026-10-16 (amendment): an undo puts the database back exactly as it was before the step ran, so rolling back a migration whose Up adopted an existing index keeps that index.

Amended 2026-10-17 (amendment): a feature flag never runs, skips or reverses a migration, and no migration reads a flag.

Amended 2026-10-17 (amendment): production data only moves forward. A rollback deploys the previous build and normally leaves the database as it is; data migrations are Irreversible by default, and index migrations keep their Downs.

Amended 2026-10-17, later (amendment): replaces the previous paragraph's guarantee that the previous build can always run against the current database. Deployments also go forward only: a rollback is a new release planned when it is needed. A compensation that would lose data runs a dry run first, reports what would be lost and exports it, and runs only after explicit confirmation. Compatibility is kept only for the rollout window.

Extended 2026-10-17 (§8): the bind mode, the environment binding record and the RequiresBinding pre-write check, the prerequisite the runtime feature releases plan sets before migrations 0009 and 0010. Those two, unique partial indexes on runtimeFeatureFlagAudit (#4272), are the first migrations to require it, so enabling the Job in any environment now needs a bind first (§8). 0011–0020, the indexes on the new runtime release collections (#4281), require it too.

Unknown-field tolerance is a backstop only

Asked about relying on old builds tolerating new fields, Chris added: "ok, but should have a system where this is never relied upon! Explicit migrations, rollbacks etc." Unknown-field tolerance ([BsonExtraElements] / ignore-extra-elements, CD plan G4a) stays, but only as a backstop. Every persisted schema change ships an explicit migration with a Down, or Irreversible with a reason, and a planned rollback target. Where the production baseline cannot read a shape and no migration covers it yet, it is recorded known debt with a named rollback target (T4). Nothing may rely on tolerance to make a rollback work. Three mechanisms enforce it. Since the forward-only amendment, a data migration is Irreversible by default; the mechanisms below are what make that safe:

  1. Migrations are required for shape changes. A PR that adds or changes a persisted field old builds cannot read must add a migration with a Down, or Irreversible with a reason (§1). The migration tests and migrations.lock.json (PmMigrations) show how.
  2. Reliance is detected, not silent. Planned, not implemented. Production counts and logs, as a metric plus a warning, every document read with unknown elements on a governed collection. This needs capture, not silent ignore: types reachable from a governed collection, nested value objects included, capture unknown elements in a form the reader can count. A non-zero count after a rollback is a defect, not a working rollback.
  3. The N-1 read check is the enforcement gate. Planned, not implemented. (Superseded as a general gate by the deployments amendment; kept only for the catch-up rollback to 104eca9.) The continuous delivery plan's old-build corpus reader (G4b; decision D3, option (b)) reads data written by main using the production build's model and fails on any shape that build cannot read.

Mechanisms 2 and 3 have numbered acceptance criteria (T2–T5) in the plan's section 8.

The delivery plan, the MVP boundary and the acceptance criteria are in the pre-deploy migrations plan. This ADR is the permanent record of the design. It implements item 7 of Phase 3 and the "migration step" that G5 points at in the continuous delivery plan.

Context

What happens today

  • API and PM build their MongoDB indexes at startup. CreateIndexesAndMapping (an IRunAtInit task in each host) calls MongoPmUnitOfWork.InitialiseIndexes(), which calls InitialiseIndexesAsync() on every repository. StudyRepository creates 17 indexes on pmStudy and ProjectRepository one on pmProject. About 30 small project-statistics collections get their indexes the same way.
  • ExecuteInitAndStartupMethods() runs before app.RunAsync(), so nothing listens on the HTTP port until every index exists.
  • Liveness has initialDelaySeconds: 60 and there is no startup probe outside preview. The preview probe (_deployment-dotnet.tpl, rendered only when preview.enabled && preview.waitForDatabase) exists because index creation on a freshly seeded preview database takes more than 60 seconds.
  • createIndexes on an index that already exists with the same specification returns at once. So startup is cheap in steady state and expensive only when a release adds an index.

What the catch-up promotion would do

Production API and PM run 104eca9. Main adds three indexes on collections that exist in production (finding A4 of the 2026-10-14 promotion audit):

Collection Index Source
pmStudy (about 10.7 GB) ProjectId_1_WorkloadShareBucket_1 #2991
pmStudy ProjectId_1_BulkUpdateLock.OperationId_1, partial on BulkUpdateLock.OperationId existing #3909, ADR-020
pmProject bulk_pdf_upload_sweep_status on BulkPdfUploadJobs.Status #2788

A partial index still scans the whole collection while it builds. The first production pod would block in CreateIndexesAndMapping for the length of the pmStudy builds. Liveness would kill it at about 60 seconds and the pod would crash-loop. The chart default deploymentStrategy: Recreate (finding A7) means the old pods are already gone by then, so production would be down.

The rollback problem

Main always writes four fields into each pmStudy.ExtractionInfo.SessionTallies[] element:

  • NumberOfSlotReservations;
  • NumberOfDirtySlotReservations;
  • TotalAllocatedSessionCount;
  • TotalEngagedSessionCount.

SessionTally is a value object with no extra-elements handling in 104eca9, so that build throws FormatException on any study main has saved (finding A3). Rolling production back to 104eca9 after the catch-up promotion therefore needs a data change first: $unset those fields.

Main mostly tolerates their absence:

  • StudyRepository.AllocationReads and the atomic capacity guard fall back from TotalAllocatedSessionCount to NumberOfCandidateSessions for "legacy tallies";
  • the tally getters recompute all four values from Sessions and SlotReservations on every whole-study save.

The exception is three filters in Filters.cs (MinimumNumberOfStageSessions, FewerThanMinStageSessions and one other). With activeReviewerTrackingEnabled on, they compare the stored TotalAllocatedSessionCount directly, so a legacy tally counts as zero allocated. The capacity guard still refuses the write, so the effect is wrong pools and counts, not over-allocation. The flag defaults to off and no environment's GitOps values turn it on, so the catch-up promotion is not affected. Before the flag is enabled anywhere with legacy tallies, those filters get the same fallback (#4182).

A backfilling Up cannot replace that fix: it would run in PreSync while 104eca9 still serves, and write exactly the fields 104eca9 cannot read. So the forward direction gets no backfill, and the rollback needs only the $unset.

Existing machinery

Machinery What it does Reuse?
PM SYRF_INDEX_INIT_MODE (Program.cs) One-shot mode: builds a reduced container, runs every repository's InitialiseIndexesAsync, exits. Used by the preview DatabaseLifecycle post-seed Job Yes, the pattern. The migration entry point is a sibling mode in the same image, but it builds no DI container
DatabaseMetadataWriter, syrf_metadata Best-effort record of service instances and lifecycle events (index_init_completed) Yes. The runner pushes migration_* events to it, and the rollback writer check reads its live instance list
IEarlyInitTask Orders init tasks that set up shared state No. Migrations must not run at app startup
Audit.SchemaVersion / DefaultSchemaVersion on aggregates A per-document stamp from configuration, used by an old in-app migration (GetAllProjectsPartiallyMigrated) No. It describes one document, not the database, and changes with configuration
SyRF.Identity.Migration A System.CommandLine campaign CLI for the Identity database, with production-safe input rules and a run store Ideas only: the run record and the "production inputs never come from arguments" rule. It targets a different database and is a separate image
StudyPendingStatisticsIndexOperation The pending-statistics index is "defined here, never built here": an administrator builds it through an endpoint, and the fold enable gate checks it exists Yes, the precedent. Index definitions in raw BSON so they do not change shape when a class map changes; the app checks rather than builds
API presync-rabbitmq-vhost.yaml An ArgoCD PreSync Job (preview only) Yes, the precedent for a native ArgoCD hook in a service chart

No general migration runner exists, so this ADR adds one.

Decision

1. The model: versioned, ordered, reversible migrations

A migration is a class with:

  • an integer Version and a stable Id (0003_pmProject_bulk_pdf_upload_sweep_status);
  • a Description;
  • a canonical Definition: the exact operation in Extended JSON (for an index, the collection, keys and options);
  • UpAsync and DownAsync, or Irreversible(reason), which means "taking the database back past this needs an Atlas snapshot restore" and says why. Since the forward-only amendment that is the normal state of a data migration: an app rollback deploys the previous build and leaves the database where it is, so it never needs to go back past one.

Rules:

  1. Ordered and append-only. Versions are 1, 2, 3 and so on with no gaps. A new migration always takes the next number. A checked-in lock file (migrations.lock.json: version, id, checksum) fails a test if an existing entry changes, is removed or a version is inserted in the middle.
  2. Immutable once merged. The checksum is SHA-256 over Id and Definition. If the database records a checksum for an applied migration that differs from the binary's, up fails. A correction is a new migration.
  3. Idempotent and resumable. Running UpAsync twice leaves the same state as once. Data migrations select only the documents that still need the change, so a re-run continues where the last one stopped. They work in bounded batches (default 500 documents by _id, with a short pause) so that a large change neither holds one long operation nor floods the oplog of an M20 cluster.
  4. Raw BSON, not class maps. A migration names fields as strings and works on BsonDocument. A class map changes when the model changes; a migration must keep doing exactly what it did when it was applied.
  5. Expand-only in the forward direction. An Up runs while the previous release is still serving (§2), so it must leave data the previous release can read. Indexes and additive backfills qualify. Anything else needs the maintenance pattern (§4) and Chris's approval. It must also leave data that every flag state of the current and previous build can use (amendment 2026-10-17).

Three operation kinds cover the MVP and most foreseeable needs:

Kind Up Down
CreateIndexMigration createIndexes with the declared name, keys and options. An existing index with the same name and specification counts as applied ("adopted"). The same name with a different specification fails (IndexOptionsConflict/IndexKeySpecsConflict), so the Job fails safe dropIndex by name, unless its Up adopted an existing index, which the runner keeps; absent counts as done. Amended 2026-10-16 (amendment)
BackfillMigration Batched updateMany over a "needs it" filter The author's reverse operation, or Irreversible(reason)
UnsetFieldsMigration A no-op, or a backfill if the fields must exist Batched $unset (including $[] paths inside arrays), then a count that must be zero

State. Collection syrf_migrations, one document per migration:

{
  "_id": "0004_pmStudy_SessionTally_reservation_fields",
  "version": 4,
  "checksum": "sha256:…",
  "status": "applied",            // applied | rolledBack | failed
  "appliedAt": "2026-…",
  "durationMs": 1234,
  "outcome": "applied",           // applied | adopted | noop
  "appliedBy": { "service": "project-management", "version": "…", "sha": "…",
                 "host": "<pod>", "actor": "argocd:<app>:<sync revision>" },
  "history": [ { "action": "up", "at": "…", "durationMs": 1234, "by": { … } } ]
}

The current schema version is the highest version with status: applied. Every Up and Down also pushes a migration_applied or migration_rolled_back lifecycle event to syrf_metadata.

A Down sets status: rolledBack and appends a down history entry. When the migration's Up had adopted the change, the runner keeps it instead of running the Down (§4): that entry carries "outcome": "kept" and the migration_rolled_back event carries kept: true. The entry of a Down that ran has no outcome. The top-level outcome keeps describing the last Up.

Lock. One runner at a time, through a lease document in syrf_migrations_lock (_id: "pm", owner, expiresAt). The holder renews it every 30 seconds for 2 minutes; another runner waits for it (default up to 60 minutes) and then re-reads the state, usually finding nothing left to do. An expired lease (a killed pod) is taken over, and because every migration is idempotent the new holder simply re-runs the interrupted one.

2. Where it runs: an ArgoCD PreSync Job in the API and PM charts, using the service image

A new syrf-common.migrations.job template renders a Kubernetes Job with argocd.argoproj.io/hook: PreSync and hook-delete-policy: BeforeHookCreation. The API and PM charts both include it, behind migrations.enabled (default false). It:

  • uses the service's own image and imagePullSecrets, with SYRF_MIGRATION_MODE=up;
  • gets only the GitVersion, runtime, logging and MongoDB environment blocks, from the same mongo-db secret as the Deployment. The application user can already create and drop indexes: startup creates them today and CreateSagaTtlIndex drops a conflicting one;
  • sets backoffLimit: 0, activeDeadlineSeconds from values (default 5400), restartPolicy: Never and automountServiceAccountToken: false;
  • passes the Helm release name and the image tag as SYRF_MIGRATION_ACTOR. Under ArgoCD's helm template, .Release.Revision is always 1, so the image tag is what tells releases apart.

Both charts render it because each Application syncs on its own. Each sync is then gated by its own Job: API's Deployment is not touched until API's Job has seen every migration applied, even when PM's Job is the one doing the work. The lease makes the second Job wait and then finish as a no-op.

Failure semantics. If the Job fails or times out, the PreSync phase fails and ArgoCD does not apply the Sync phase. The Deployment is unchanged, so the old pods keep serving the old build. The production sync policy retries up to three times; each retry deletes and recreates the Job, and the runner resumes. After the retries ArgoCD reports the sync as failed and does not retry that revision again on its own. Fixing it means a new cluster-gitops commit: a fix, or the toggle turned off.

PreSync hook, not a Helm pre-upgrade hook. ArgoCD renders charts with helm template and maps Helm hooks onto its own phases, so either would run. The native annotation is chosen because:

  • it is what the chart already uses (presync-rabbitmq-vhost.yaml, _postsync-notify.tpl);
  • its delete policy and phase names are the ones operators see in the ArgoCD UI;
  • nothing deploys these charts with helm upgrade (GitOps only), so Helm's own hook semantics would never be exercised.

Service image, not a dedicated migrator image. A separate image would be another build, another tag to promote and a way for migration code to drift from the app build it serves. The service image already carries the PM data library, and the existing SYRF_INDEX_INIT_MODE proves the one-shot pattern. The migrations live in SyRF.ProjectManagement.Mongo.Data, which both images contain.

Version skew. API and PM are versioned and promoted separately, so their images can hold different migration lists. Because the list is append-only and every binary applies it in order, the database always holds a prefix of the newest binary's list. A binary that finds applied migrations it does not know (the database is ahead of it) leaves them alone and succeeds; it never runs a Down on its own.

3. Long index builds: createIndexes from the Job, not the Atlas rolling build

createIndexes from the PreSync Job Atlas rolling index build (Admin API)
How The driver sends createIndexes to the primary. Since MongoDB 4.4 every data-bearing member builds at the same time and commits together (default commit quorum: all voting members). The build takes an exclusive collection lock only at its start and end; reads and writes continue between Atlas takes one member at a time out of the replica set, builds the index on it as a standalone, and returns it
Load All three M20 members (2 vCPU each) scan the collection at once, using up to maxIndexBuildMemoryUsageMegabytes (200 MB by default) and spilling to disk. Expect higher latency during the build One member at a time, so the primary is never loaded by the build
Availability Full replica set throughout One member down at a time: a three-member M20 has no spare while each member is out
Duration One pass per member, in parallel Roughly three passes back to back
Credentials The application's existing database user A programmatic Atlas Admin API key stored in the cluster, with a project role that can manage indexes: a new and powerful secret
Control Synchronous: the Job ends when the index is ready, with the server's own error on failure Asynchronous: the Job must poll the Atlas API, which has its own failure modes
Testability Identical in Testcontainers, preview, staging and production Cannot run in Testcontainers or the local stack

Recommendation: createIndexes from the Job. On the repository's own evidence, index builds on M20 over a collection of about 3.1 million documents took 2 to 5 minutes and did not block queries (index cleanup commands). A build of that size is well within a PreSync Job's deadline while the old pods serve. The rolling build trades a short period of higher load for a longer period without redundancy, plus a new Admin API secret, and it cannot be tested.

Two safeguards make the choice reversible:

  • Rehearsal first. Timing and latency are measured before production, on a cluster restored from a production snapshot (plan R2). If they are unacceptable, the plan switches to the fallback.
  • Fallback: build out of band, then adopt. An operator builds the index with Atlas's rolling build in the Atlas UI before the promotion. The migration's Up then finds the index with the same name and specification, records outcome: adopted, and the Job finishes at once. No code changes.

Retries during a build. The index build runs on the server and is not tied to the Job's connection. If the Job pod is killed mid-build, the retried Job sends the same createIndexes. The expected behaviour on MongoDB 4.4 and later is that the server attaches it to the build in progress rather than starting a second one; plan R2 verifies this on Atlas before production relies on it.

Batching. Each index is its own migration, so each can be rolled back on its own. Two createIndexes calls on pmStudy mean two collection scans. If the rehearsal shows that matters, a later migration can build both in one command; the MVP keeps them separate.

4. Rollback: Down migrations to a target version, then the old image

SYRF_MIGRATION_MODE=down with SYRF_MIGRATION_TARGET=<version> runs DownAsync for every applied migration above the target, newest first. It refuses to start if any of them is Irreversible (the error names the migration and its reason) or if the target is above the current version.

A Down puts the database back as it was before the Up ran. So a migration whose Up adopted an existing change (outcome: adopted) is rolled back without running its Down: the change stays, the migration is marked rolledBack (the next Up runs it again and adopts again), and the history records kept. The refusals still apply first: an Irreversible migration refuses the Down even when its Up adopted. Amended 2026-10-16 (amendment).

A Down that changes data races with the new build, which may write the fields again while the Down runs. So rollback is a maintenance procedure of two cluster-gitops PRs, both approved by Chris:

  1. R-1, "drain and down". API and PM stay on the new chart and image, with replicaCount: 0 and migrations.rollback.targetVersion: <N>. The Sync phase scales both to zero. A PostSync Job (the same template, mode down) then runs the Down migrations. Before changing data it waits until syrf_metadata lists no live syrf-api or syrf-projectmanagement instance, so it cannot start while the other Application's pods are still writing. A pod removes its entry on graceful shutdown (DatabaseMetadataWriter.WriteDisconnect); a crashed pod's entry stays, so the wait has a timeout and an explicit migrations.rollback.skipWriterCheck override for an operator who has confirmed in ArgoCD that both Deployments are at zero. The Job finishes by checking that no document still matches the Down's filter.
  2. R-2, "old build". API and PM go back to the old chart and image tags with their replicas restored. The old chart ignores the migrations.* values.

Production is unavailable from R-1's sync until R-2's pods are ready. That matches today's Recreate outage and is the price of a data rollback. Every Down runs through R-1, including an index-only Down. A rollback that needs no Down at all skips R-1: R-2 alone deploys the old build, and the migrations stay applied because their changes are compatible with it (rule 5 of §1).

Choosing the target. The target is the highest version the old build can live with, not automatically zero. An extra non-unique index is invisible to an older build, and dropping it would force a full rebuild at the next promotion. So the catch-up rollback to 104eca9 targets version 3: it undoes the SessionTally migration (version 4) and keeps the three indexes 1–3 that 104eca9's collections carry. A Down to version 3 also crosses every migration added after version 4 (in reverse order): 20 down to 11, which index the new runtime release collections (runtime feature releases §6.4), then 10 and 9, the unique partial indexes on the small existing runtimeFeatureFlagAudit collection (runtime feature releases; the first migrations to require the binding of §8), then 8, 7, 6 and 5, which index the new pmDeletionOperation and pmDeletionLedger collections (ADR-024). Nothing builds an index on any of those collections at startup, so each of these Downs drops an index its Up created, unless the Up adopted one built out of band, which is kept.

The rule, pinned by PmMigrationsLockFileTests (amended 2026-10-16, see the amendment): a Down restores the state before its Up, so an index the Up created is dropped and an index the Up adopted is kept. Every migration after version 4 that a target-3 rollback crosses must therefore be an index migration (CreateIndexMigration). Dropping an index the step created leaves the old build exactly the database it ran on before. Three caveats:

  • The forward rule still applies. An Up runs while the previous release is serving (rule 5 of §1), so a unique index must never be one the previous release could violate.
  • A large collection costs a rebuild. An index migration after version 4 on a large collection (pmStudy is about 10.7 GB in production) is safe to cross, but its Down drops the index and forces a full rebuild at the next promotion. Weigh that when choosing the target.
  • A drop-index kind would recreate. The Down of a future drop-index migration kind recreates the index it removed. No such kind or migration exists today.

A later migration of any other kind needs this rule, and the rollback target, revisited before it merges. Index Downs still exist, are tested, and are used when an index itself is the problem.

The first real Down. Migration 4, 0004_pmStudy_SessionTally_reservation_fields:

  • Up is a no-op recorded as noop. Main writes the fields itself and tolerates their absence.
  • Down unsets the four fields with $[] paths, in batches over the documents whose tallies contain any of them, then requires a count of zero.

After the Down, Study.SlotReservations (a root field) is left in place. 104eca9 has no slot reservations, so it keeps the field as an Entity extra element but never maintains it. The scheduled clean-up (RemoveSuspendedSession, the connection liveness checks) does not exist in that build. If production later rolls forward again, main's getters recompute the four tally fields on the next save of each study, and any reservation left from before the rollback would count as a held slot until main clears it. Reservations exist only while activeReviewerTrackingEnabled is on, and that flag is off in every environment's GitOps values today. Whether the rollback Down should also remove SlotReservations (all reviewers are disconnected by the drain anyway) is plan D10.

Rehearse before relying on it. The full Up, R-1, R-2 and Up-again cycle is rehearsed on staging (plan R1 and R3), rolling back to the production baseline build itself, and the timing is measured on a restored production snapshot (plan R2). Nothing runs against production without Chris's approval of the specific cluster-gitops PR.

5. App side: check, do not build

A deployment setting Migrations:PreDeploy (environment SYRF__Migrations__PreDeploy), rendered from the same migrations.enabled value that renders the Job, changes three things when it is on:

  1. Startup skips migration-owned indexes. A StartupIndexPolicy service tells StudyRepository and ProjectRepository which index names a migration owns. With PreDeploy off it owns none, so startup behaves exactly as today.
  2. Readiness waits for the schema. A health check tagged ready and schema reports Unhealthy until every migration this binary knows is applied with a matching checksum and every index those migrations declare exists with the declared specification (a listIndexes per governed collection). A pod in this state is running but not ready: it receives no traffic and is not restarted. /health/live excludes schema-tagged checks, so liveness cannot kill a pod for a missing migration. The check is a startup gate only. After its first healthy result it re-checks every 5 minutes and reports a mismatch or a failed read as Degraded, which keeps the pod ready but visible. A transient Mongo error, or an index dropped by hand, must not take every pod out of service at once.
  3. Startup never builds a governed index. An architecture test (§6) enforces it.

Why a readiness gate and not a crash. A crash-looping pod hides the cause behind restarts and back-off. A not-ready pod keeps its logs, shows schema: Unhealthy with the missing migration in /health/ready, and becomes ready by itself once an operator applies the migration. With RollingUpdate the old pods keep serving meanwhile.

Why a deployment setting and not a runtime feature flag. The app mode and the Job must change together: an app that skips index builds with no Job to run them would serve without its indexes. Rendering both from one chart value makes that mismatch impossible, and a runtime override must not be able to undo it. That is why it is not in the env-mapping.yaml flag catalogue.

Which collections are governed. pmStudy and pmProject: the two existing production collections that main adds indexes to. The project-statistics collections and the other new collections keep creating their indexes at startup. They are absent or empty in production, so their builds are instant, and moving them would make every fresh database (tests, local, E2E, previews) depend on a migration run for no benefit. A collection joins the governed list when it holds enough production data that a build is slow; the plan records the sizes found in the rehearsal and proposes the threshold (D7).

Startup probe. _deployment-dotnet.tpl renders a startupProbe for every .NET Deployment when startupProbe.enabled is true, with its numbers in a startupProbe: values block. The default off keeps today's production manifests unchanged. Preview with waitForDatabase keeps rendering exactly today's probe (failureThreshold: 90). The proposed production setting is failureThreshold: 30 (about five minutes), set per environment in cluster-gitops. The probe covers ordinary slow starts (container pull, class-map registration, small-collection indexes); it does not make a long index build at startup acceptable.

6. Testing

Layer What it proves Where
Unit, per migration Definition and checksum are stable; Up and Down build the intended commands; Irreversible refuses SyRF.Mongo.Common.Tests (Migrations/); PM data tests for the PM migrations
Runner unit tests Ordering, the append-only lock file, checksum mismatch, refusing Down past Irreversible, version skew, exit codes SyRF.Mongo.Common.Tests
Testcontainers integration On seeded data: Up, then Down, then Up again. Asserts index specifications from listIndexes, document shapes (the four tally fields present, absent, then present again after a save), state documents, an adopted pre-existing index (kept by a Down), a conflicting index failing, the lease (two runners, one waits), resuming after a simulated interruption SyRF.ProjectManagement.Mongo.Data.Tests (replica-set fixture)
Definition parity Each migration-owned index equals what the repository creates with PreDeploy off (name, keys, options), so the two definitions cannot drift PM data tests
Architecture (G5 code half) With PreDeploy on, the startup index pass creates no index on a governed collection outside StartupIndexBaseline (the indexes that predate the migration system, which only shrinks). A fixture repository adding a new governed index fails PM data tests
App gate The schema check is Unhealthy before Up and Healthy after, then Degraded (never Unhealthy) on a later mismatch or read failure; /health/live stays 200 throughout API and PM endpoint tests
Chart render migrations.enabled: false renders no Job and no setting (production manifests unchanged); true renders a PreSync Job with the right image, environment, deadline and delete policy, and the setting on the Deployment; rollback values render the PostSync down Job; startupProbe off by default, preview unchanged helm unittest in the API and PM charts, test-charts lane
Environment binding (§8) Fingerprint and environment grammar; bind writes, no-ops and refuses with exact messages; up refuses a run with a pending binding migration on each mismatch, writing nothing; status stays read-only and reports a hello failure instead of failing; existing checksums unchanged; on the real catalogue, up without a binding stops before 0009 with exit 2 and applies nothing (0001–0008 included), and a bound database at version 8 applies exactly 0009 onwards; a database at version 10 whose Job renders no binding values stops before 0011, building nothing, and applies exactly 0011–0020 with them; the chart renders the values only when set SyRF.Mongo.Common.Tests (Migrations/EnvironmentBinding*), PmMigrationsLockFileTests, PmMigrationsBindingTests, PmRuntimeReleaseIndexMigrationsTests, migrations_binding_test.yaml in both charts
Forward rule for a unique index (rule 5 of §1) The previous release's writers never produce a document inside the partial filter: for 0009–0010, today's audit documents carry neither mutationId nor storeEpoch; legacy rows that share a revision insert and survive the index build; today's store keeps writing with both indexes applied. Each with a same-data positive control (duplicate key 11000). For 0011–0020, which index new collections: no production source but the catalogue names them (positive control: the same scan finds the audit collection's writer), and the unique event and outbox indexes reject with 11000 a pair they accepted before the migrations SyRF.API.Endpoint.Tests (RuntimeFeatureFlagAuditDocumentShapeTests, RuntimeFeatureFlagAuditIndexForwardRuleTests); PM data tests (RuntimeReleaseCollectionsForwardRuleTests, PmRuntimeReleaseIndexMigrationsTests)
TTL index (0020, OD-20 retention) The server removes a report once its deleteAfterUtc has passed, and keeps one with a later time, one without the field and one with a non-date value PM data tests (PmRuntimeReleaseIndexMigrationsTests)

7. Flag decision

Flagged, by the deployment toggle migrations.enabled, default off in the chart and in every environment. Off renders no Job and leaves Migrations:PreDeploy false, so startup creates every index as it does today and production manifests do not change. Chris turns it on per environment in cluster-gitops: staging first, then production in the catch-up promotion PR. Turning it off again is the kill switch; the app then builds any missing indexes at startup, as today.

8. Environment binding

Added 2026-10-17 (#4258), the prerequisite §6.3 of the runtime feature releases plan sets before migrations 0009 and 0010. An environment name is not an authority: a missing values file renders a chart default, staging and previews share one cluster, and a restored snapshot carries the source's metadata. So a database is bound explicitly, once, to an environment, a database name and a cluster, and a migration that must only run against the right database says so.

The record. Collection runtimeEnvironmentBinding, one raw-BSON document (_id: "binding"), read and inserted with majority read and write concern:

Field Value
environmentId production, staging, preview:pr-{n} (no leading zero) or local:{stack} (a lower-case DNS label); nothing else is accepted
databaseName The configured database
clusterFingerprint The live cluster fingerprint at bind time (below)
bindingEpoch A new GUID per binding, stored as CSUUID like every SyRF GUID
boundAtUtc, boundBy When, and who: the runner's actor, as in appliedBy
announcementDelivery Derived from the environment, never an input: production for production, testSink for staging, disabled for preview:* and local:*

The cluster fingerprint. One canonical string from the live hello command, compared exactly. Operators put it in chart values; status prints the live one to copy.

  • Replica set: <setName>@<hostSuffix>, for example atlas-abc123-shard-0@abcde.mongodb.net. The host suffix comes from hello.hosts (the electable members; passives, arbiters and hidden members are left out), each without its port, lower-cased and without a trailing dot. One distinct host gives that host (a single-member test replica set: rs0@<container host>). Several give their longest common DNS suffix in whole labels, so Atlas's cluster0-shard-00-0N.abcde.mongodb.net members give abcde.mongodb.net, the same through the -pri private horizon. Two Atlas clusters in one project share the suffix and differ by set name. The suffix is kept only when it has at least two labels (#4263): hosts that share no label, or only a bare TLD such as com, are listed, sorted and comma-separated (rs0@mongo1,mongo2, rs0@a.example.com,b.other.com). An IP address is one label, never split on its dots, so several IP members always give the list.
  • Standalone (no setName): standalone. Its hello names no host; standalone servers exist only in local and test stacks, where the database name and the record still apply.
  • Refused: a mongos, a replica-set member without a configuration, and a replica set reporting no hosts.

SYRF_MIGRATION_MODE=bind. Requires SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT, _DATABASE and _CLUSTER, and checks, in order:

  1. the configured RuntimeEnvironment is the one the expected environment runs as: the part before the colon, so preview:pr-4258 runs as preview and local:e2e as local (compared exactly, case-sensitive);
  2. the configured database name equals the expected database;
  3. the live fingerprint equals the expected cluster.

With no record it inserts one (insert-if-absent on _id, so two concurrent binds cannot both win; the loser compares with the winner). An identical binding (same environment, database and cluster) is a no-op that keeps the original epoch. A different binding is refused, never overwritten. A new binding pushes an environment_bound lifecycle event to syrf_metadata. bind takes no lease: it writes one document atomically.

RequiresBinding. IDatabaseMigration.RequiresBinding, false by default. When the pending set of an up run contains such a migration, the runner, under its lease and before applying anything in that run (pending migrations that do not require the binding included), runs checks 1–3 and a fourth: (4) the binding record exists and matches all three. A run with none pending never reads the variables. A migration that requires the binding declares requiresBinding: true in its Definition, so the checksum covers it; the key is emitted only when true, so every existing definition and checksum is unchanged, and MigrationCatalog.Validate refuses a flag the definition does not declare. CreateIndexMigration takes it as an optional constructor argument. No migration up to 0008 requires it; 0009 onwards are the first.

status prints the binding (or its absence) and the live fingerprint and, when any expected variable is set, every mismatch as a warning. It writes nothing and exits 0. A driver error or timeout from hello does not fail it (#4263). It prints the live fingerprint as unavailable, with the reason, and lists that reason as a mismatch when expected values are set. bind and up still fail on it.

Exit codes. Every binding refusal is exit code 2 (usage) and writes nothing: a missing or invalid expected value, each of checks 1–4, and a different existing binding. For up that means no syrf_migrations record, no index and no partial apply; the lease is released.

Chart. migrations.binding.expectedEnvironment, .expectedDatabase and .expectedCluster are rendered into every migration Job as SYRF_MIGRATION_BINDING_EXPECTED_*, each only when non-empty (they cannot come through env:). migrations.binding.bind: true, with migrations.enabled and all three values, adds a PreSync bind Job in sync wave -1, ahead of the up Job; rollback mode still renders only the down Job. Everything defaults to empty, which renders exactly what it did before.

When the fingerprint changes (operator guidance). The fingerprint comes from the live hello.hosts. So it changes when a replica set is moved, rebuilt under new hostnames, or restored into a new Atlas project; a new connection horizon (-pri against public) does not change it.

  • The next up with a RequiresBinding migration pending then refuses with exit 2 and writes nothing. The message names "the live cluster fingerprint '…' does not equal the expected cluster '…'", or the stored binding that no longer matches.
  • That is the intended, fail-closed symptom, not a fault in the runner. status prints both values.
  • An up with no such migration pending is unaffected.
  • The remedy is rebind (runtime feature releases plan §6.3, point 4), which is not built yet and is a follow-up. Until it exists no supported command replaces a binding, because bind refuses a different one by design. Raise the case with Chris rather than editing the record by hand.

First users and the local stack. Migrations 0009 and 0010 (#4272) are the first to require the binding.

  • Deployed environments. Enabling the Job in an environment needs a binding first. Either render the bind Job in the same change (migrations.binding.bind: true with the three expected values), or run bind before. Otherwise up stops before 0009 and applies nothing.
  • The local E2E stack. It binds itself in e2e/scripts/pm-migrations.sh, with RuntimeEnvironment=local. status supplies the live fingerprint, which the binary computes. bind then binds to local:<stack>, and up follows. This adds no code path in the binary that accepts an unverified cluster, and nothing changes for production or staging.

Not yet. These are still to come:

  • rebind for restored copies;
  • binding checks on policy and release writes (runtime releases increment C);
  • the preview database lifecycle Job binding automatically, once previews run up (today the migration Job refuses preview.waitForDatabase).

Three questions are open (#4263):

  • whether down should check the binding;
  • whether status should exit non-zero on a mismatch;
  • whether migrations.binding.bind should stay set after the first bind.

Flag decision. Not flagged: the mode does nothing unless an operator runs bind, and the check runs only when a migration that requires the binding is pending. Neither happens by default.

Consequences

Positive.

  • A promotion that adds an index on a large collection builds it while the old build still serves, and a failure leaves production on the old build instead of crash-looping.
  • Every schema change has a recorded state, an explicit rollback and an owner-approved procedure.
  • The catch-up promotion's A3 and A4 findings get a mechanical, rehearsed answer.
  • G5's code half gains a positive rule ("put it in a migration") next to its prohibition.

Negative.

  • Each new index on a governed collection is now a migration plus a parity entry, not one line in a repository.
  • A data rollback is a two-PR maintenance procedure with downtime.
  • The PreSync Job lengthens every sync by the Job's start-up time (seconds) when there is nothing to do.
  • The application's database user keeps createIndex and dropIndex. A separate migration user with the application reduced to read/write is a possible follow-up, not part of this decision.

Alternatives considered

Alternative Why not
Keep building at startup, add a long startup probe The pod still serves nothing during the build, Recreate still means an outage, and a failure still crash-loops. The CD plan's G5 already rejects it
Build in the background after Kestrel listens The app serves without the index (slow queries on a 10.7 GB collection) and nothing records whether the build finished. No rollback story
A runbook only (mongosh before promotion) Not tested, not recorded, easy to forget; the opposite of "automating is def preferable"
Off-the-shelf runner (Mongock, migrate-mongo) Mongock is Java and migrate-mongo is Node: another runtime in the image or a separate image. The runner here is small, and the index and batching rules are SyRF-specific
Rely only on reader tolerance (CD plan G4a) Protects rollbacks to builds that contain the convention, not to 104eca9. Chris asked for explicit migrations. G4a still goes ahead, as a backstop only (Status)
A compatibility image (104eca9 plus SessionTally tolerance) Removes the need for the Down for this one rollback, but every later incompatible change would need another image. It remains a fallback if the Down rehearsal fails (plan D4)

Amendment 2026-10-16: an undo keeps an adopted change

Decision. OD-10 option C, decided by Chris on 2026-10-16 in the runtime feature releases decision sheet (see the runtime feature releases plan): an undo must put the database back exactly as it was before the step ran.

Why. CreateIndexMigration's Up records outcome: adopted when an identical index already exists (built out of band, as in the §3 fallback, or by startup before the Job was enabled). Its Down dropped the index by name all the same, so undoing an adopted step left the database with less than it had before the step ran, and an index that predated the migration had to be rebuilt in full.

What changed.

  • The runner decides, for every migration kind. When an applied migration's recorded outcome is adopted, MigrationRunner.DownAsync does not call its DownAsync. It still marks the migration rolledBack, so the schema version drops and the next Up runs it again, which adopts again. The down history entry records "outcome": "kept" and the migration_rolled_back event carries kept: true (§1, State). A record without an outcome runs its Down as before.
  • Refusals are unchanged and come first. An Irreversible migration above the target refuses the Down even when its Up adopted: irreversibility belongs to the migration, not to one run of it.
  • A misclassification only ever keeps an index. If an Up built an index and its pod died before recording it, the retried Up adopts the index it built, and an undo keeps it. That is the safe direction: the Up ran while the previous release was serving, and rule 5 of §1 means a migration never adds an index, a unique one above all, that the previous release could trip over.
  • The catch-up rollback rule is simpler (§4, "Choosing the target"). Every migration after version 4 must be an index migration; it no longer has to sit on a migration-only collection. PmMigrations.MigrationOnlyCollections stays, but only for start-up index governance (§5): the indexes on those collections are defined as migrations and never built at startup.

Consequences. Rolling back an adopted index costs nothing and leaves the old build the database it ran on. An index migration after version 4 may target an existing collection, provided rule 5 holds; on a large collection its Down still forces a rebuild at the next promotion, which weighs on the choice of target.

Amendment 2026-10-17: flags are not migrations

Decision. Chris, 2026-10-17: "Flags are used to turn features on and off, not necessarily to auto migrate databases back and forth. So whilst the flag is present and flippable both the database and app code should work well with either on or off state and should be happy going back and forth."

Why. Four terms were getting confused: environment, deployment, migration and flag. Left unstated, that invites a flag to be treated as the way to prepare or undo data, or a migration to be sequenced around a flag's value. They are four different things; the definitions are in Environments, deployments, migrations and flags.

What changed.

  • A flag never runs a migration. A feature flag switches behaviour in a running build. It never runs, skips or reverses a migration, and no migration reads a flag.
  • A migration suits every flag state. A migration must leave data that every flag state of the current and previous build can use. Rule 5 of §1 (expand-only in the forward direction) already requires this of the previous build; it now covers each flag state of both.
  • The only allowed link runs the other way. A feature may refuse to switch on until its migrations have run (fail closed). The schema readiness gate of §5 has the same shape: the feature stays closed until the schema exists, and nothing about the flag applies the schema.
  • Round trips are the flag author's job. Code and data must survive the flag going on, off and on again, and the extra code, collections and indexes that exist only for the flag are removed when the flag is retired, by a migration where indexes or data are involved. The requirements are in Requirements when a flag is added, with the agent checklist in .claude/rules/feature-flags.md.

Consequences. This amendment changes no code: the runner, the charts, the lock file and every existing migration are as they were. It adds a requirement for new migrations (every flag state of both builds can use what they leave) and for new flags (the round-trip requirements above).

Amendment 2026-10-17: production data only moves forward

Superseded in part by the deployments amendment below: the previous build no longer has to run against the current database, so the expand-then-contract window, G4a, T2, T2a and G4b below no longer apply as general requirements. They remain only for the catch-up rollback to 104eca9.

Decision. Chris agreed on 2026-10-17 that production data only moves forward. A rollback deploys the previous build and leaves the database as it is, so the previous build must work with the current database. Putting the database itself back to an earlier state means restoring a backup, which loses every write since: that is disaster recovery, not rollback.

Why. Users keep working between a promotion and a rollback. A data Down written in advance has to guess what that work will look like, and it races with it, which is why every Down needs the R-1 outage (§4). A forward fix written once the problem is known can account for what has happened since. Index Downs have neither problem, because an index holds no user data.

What changed.

  • An app rollback normally needs no database change. R-2 alone (§4) deploys the previous build. The exceptions are the catch-up rollback below and, until G4a is in production's rollback target, an approved T1 Down for a nested field.
  • A Down only reaches back to the newest Irreversible migration. down refuses to cross an Irreversible migration (§4, plan M7), and that refusal stays: a Down to a target must undo everything above it, and skipping a data step in the middle would leave a database no build was tested against. So R-1 can still undo an index migration newer than every data migration, but an index problem older than a forward-only data migration is fixed forward, by a new migration. A drop-index kind would be added for that if it is ever needed; none exists today.
  • Data migrations are Irreversible by default. A BackfillMigration, or any later data kind, declares Irreversible with the reason "forward-only (ADR-023 amendment 2026-10-17)", unless its PR argues for a Down and Chris approves it. A data change that turns out to be wrong is fixed by a new, compensating migration.
  • Index migrations keep their Downs. They are tested, and they are used when an index itself is the problem, within the limit above. The 2026-10-16 amendment still applies: an adopted index is kept.
  • Expand, then contract. (Superseded by the deployments amendment: compatibility is kept only for the rollout window.) Rule 5 of §1 already makes every Up expand-only while the previous release serves. A contract step (removing or renaming a field, dropping a collection) ships only once no build older than the expand step can come back, which means once the expand build is production's rollback target. Until then, the new build reads and writes both shapes, or the new field is recomputable or unused (the CD plan's G4a rule).
  • The previous build keeps fields it does not know, and that is proven, not assumed. (Superseded by the deployments amendment; G4a is on hold.) Root documents keep unknown fields through Entity.ExtraElements. Nested value objects such as SessionTally do not, and that is the only reason migration 4 needs a Down. CD plan G4a (capture unknown elements on every persisted type, nested ones included, with a coverage test) and the pre-deploy plan's T2 and T2a close the gap. The policy applies in full only once production's rollback target contains G4a. Until then, a field added to a nested type the previous build cannot read still needs a Down under T1, as an approved exception. This keeps the 14 October rule that tolerance is never relied on unverified: forward-only depends on previous-build compatibility being tested (G4a), counted in production (T2) and enforced by the N-1 read check (T4, CD plan G4b).
  • Flags release; deployments deploy. New behaviour ships dark behind a flag (amendment), so most problems are handled by switching the flag off, with no deployment at all.
  • Backups and point-in-time restore are for disasters only.

The catch-up exception. The catch-up promotion to main predates these rules. Its planned rollback to 104eca9 (§4, target version 3) still runs migration 4's Down and stays as planned. Once the catch-up promotion has been accepted in production, production's rollback target moves forward to that build. From then on no planned rollback crosses a data Down; migration 4's Down stays in the catalogue only because merged migrations are immutable (rule 2 of §1).

Consequences. This amendment changes no code: the runner (including its refusal to cross an Irreversible migration), the charts, the lock file and every existing migration are as they were. New data migrations are Irreversible by default, and index migrations are unchanged. R-1 becomes rare, and after the first forward-only data migration it can only undo index migrations newer than that one. Whether a contract step's expand build is production's rollback target is checked in review for now. It becomes checkable once the production compatibility report records the applied version of each service (runtime feature releases plan §7.8), which is the same source the N-1 read check uses for its baseline (CD plan G4b). The weight moves to previous-build compatibility, so G4a, T2, T2a and G4b are what make the policy safe. G4a is tracked in #4282.

Amendment 2026-10-17: deployments go forward only

Decision. Chris, 2026-10-17, on the decision page (option A), after raising it himself: keeping every earlier build able to run against the current database "adds significantly more complexity but for hypothetical problem scenarios that are by definition exceptional", loosens schema enforcement, and is a plan better made "at that hypothetical time". This amendment replaces the "previous build must work with the current database" guarantee in the forward-only amendment. Data still only moves forward. So do deployments.

What changed.

  • A rollback is a new release, planned when it is needed. It is the code we want back plus, where the data no longer suits that code, a compensating migration written for the situation as it then is. Nothing has to prepare a rollback in advance.
  • A compensation that would lose data is explicit. It runs a dry run first, which lists exactly which documents and fields would be dropped or reshaped and exports them somewhere durable. It runs only after explicit confirmation by whoever is doing the rollback. The export is what a later release can put back.
  • Compatibility is kept only for the rollout window. An Up still runs while the previous release is serving (rule 5 of §1), and the API and Project Management can briefly run different builds against the same database. So within one release a change only adds; a later release may rename, restructure, remove and validate again. The previous build is not expected to survive any database after the rollout has finished.
  • What is no longer required: expand-then-contract across releases kept "until the expand build is production's rollback target", the old-build read check as a merge gate (CD plan G4b) and capture of unknown fields on every type for rollback's sake (G4a, #4282, on hold). Whatever the rollout window itself needs is decided in #4282.
  • What stays: the migration runner, the environment binding (§8), the lock file and the two-step maintenance procedure (§4). They are how a compensating migration runs. Index migrations keep their Downs. The catch-up promotion's planned rollback to 104eca9, with migration 4's Down, stays as the one exception that was already built.
  • Flags are unaffected. Switching a flag off and on remains the routine way to stop a misbehaving feature, under the flags amendment.

Consequences. No code changes. Releases stop paying a permanent compatibility cost, and the data model can tighten over time. A real rollback costs more effort when it happens: a release and possibly a compensating migration, designed then.

References