JSON serialization contract (Newtonsoft → System.Text.Json parity)¶

The API's JSON wire (MVC responses and request bodies, and the NotificationHub protocol) is written by Newtonsoft.Json today. The System.Text.Json migration plan moves it to System.Text.Json (STJ) behind startup switches. This page is the contract that migration is measured against: what today's bytes are, how they are pinned, which differences are known and which are accepted.

Status (R0, #4145): the parity harness (PR-0.1) and 13 of the 14 STJ attribute twins (PR-0.2) exist. Nothing serialises differently: R0 adds tests, golden files and attributes that only STJ reads, and the STJ paths that already exist (swagger.json generation, MassTransit messages, the definition-metadata serialisers) produce byte-identical output. No STJ serializer is wired to the API.

Where it lives¶

Everything is in src/services/api/SyRF.API.Endpoint.Tests/Serialization/ unless noted.

Piece	File	Plan criterion
Today's wire, reproduced	`NewtonsoftWire.cs`	§4 step 4
Wire type enumeration (ApiExplorer + hub)	`WireTypeCatalog.cs`	§4 step 1
Deterministic instances	`DeterministicInstanceBuilder.cs`	§4 step 3
Newtonsoft goldens	`WireGoldens.cs`, `NewtonsoftGoldenTests.cs`, `Goldens/rest/`, `Goldens/hub/`	0.1.1, C1
`swagger.json` cross-check	`SwaggerSchemaCrossCheckTests.cs`	0.1.2
Contract scanner and allowlist	`ContractScanner.cs`, `ContractScannerTests.cs`, `contract-allowlist.json`	0.1.3, C2
ProblemDetails with a `status` extension	`ProblemDetailsGoldenTests.cs`, `Goldens/problem/`	0.1.4
Submission-digest pins (post-binding)	`SubmissionDigestPostBindingPinTests.cs`, `Fixtures/session-submissions.json`	0.1.5
Submission-digest pins (domain, screening)	`SyRF.ProjectManagement.Core.Tests/ProjectStatistics/ProjectStatisticsSubmissionDigestPinTests.cs`	0.1.5
Definition-metadata goldens	`DefinitionMetadataGoldenTests.cs`, `Goldens/definition/`	0.1.6
`JsonConvert.DefaultSettings` guard	`NewtonsoftDefaultSettingsGuardTests.cs`; runtime asserts in `ProjectStatisticsHostRegistrationTests` (PM) and `QuartzServiceIntegrationTests`	0.1.7
MassTransit contract goldens	`MassTransitContractGoldenTests.cs`, `Goldens/masstransit/`	0.2.2
Promotion safety (no existing STJ path reaches a twin)	`StjPromotionSafetyTests.cs` (API and `SyRF.PdfAgent.Tests`), walker `src/testing/StjTwinReachability.cs`	R0

Run the harness with:

nice dotnet test src/services/api/SyRF.API.Endpoint.Tests --filter "FullyQualifiedName~SyRF.API.Tests.Serialization"
nice dotnet test src/libs/project-management/SyRF.ProjectManagement.Core.Tests --filter "FullyQualifiedName~ProjectStatisticsSubmissionDigest"

Goldens¶

What is enumerated. ApiExplorer over the real controller set, with NSwag's registration so it sees the same 215 operations swagger.json describes. Every response type and [FromBody] type is normalised (task, ActionResult<T>, nullable and collection wrappers removed; dictionaries kept) to one REST golden per type. A JsonPatchDocument<T> body is goldened as itself, although ApiExplorer reports it as Operation[]. The hub side is one golden per INotificationHubClient message and per public NotificationHub method (invocation frame, plus the completion frame when the method returns a value).
How bytes are produced. REST bytes come from executing an ObjectResult against a container built like the API's (AddControllers().AddNewtonsoftJson() plus ConfigureJsonOptions), so the framework's ProblemDetails converters and formatter selection apply. Hub frames come from NewtonsoftJsonHubProtocol with a copy of Program.cs's AddNewtonsoftJsonProtocol settings. Two tests keep the copies honest: one compares the Program.cs block text, and one builds the real API host and compares its resolved MVC and hub settings with the harness's.
Instances. Members are exactly what Newtonsoft's contract resolver serialises, filled from a counter. Three variants per type: populated (UTC dates, defined enums, whole doubles, decimals with a trailing zero), edge (unspecified-kind dates, undefined enum integers, fractional doubles, escaping-sensitive text) and nulls (every member the code declares nullable set to null). Collections of a polymorphic element hold one instance of each concrete subtype; single polymorphic members rotate through the subtypes; polymorphic roots get one variant per subtype. Dictionaries use the keys ABCKey, Species name, 1a and camelCase.
Format. Each file is JSON: a subject and a bytes object whose values are today's exact output, one variant per line, so a diff shows which variant moved.
Changing them. A golden diff is a wire change. Regenerate deliberately with SYRF_UPDATE_JSON_GOLDENS=1 (it also rewrites the allowlist's finding details and deletes stale goldens) and explain the diff in the PR. A new endpoint or DTO needs its golden committed.

Contract scanner¶

ContractScanner walks the closure of every wire root and reports behaviours that differ between the two serializers unless the type opts in. contract-allowlist.json must list exactly the current findings, each with disposition: fix (and by: the plan PR that fixes it) or disposition: intentional (and a reason).

Category	Finding	Why it matters
B1	A member typed as an abstract class or interface (or a collection or dictionary of one)	Newtonsoft writes the runtime type; STJ writes the declared type and drops subtype members
B6	A request type with no constructor STJ can use	Newtonsoft falls back to a non-public parameterless constructor; STJ throws
B7	A request type's collection with no public setter or constructor parameter	Newtonsoft populates it in place; STJ leaves it empty
B8	A serialised public field, or `[JsonProperty]` on a non-public member	STJ ignores both without `[JsonInclude]`
J4	A Newtonsoft attribute in any SyRF assembly without its STJ twin	The member's behaviour would change on the switch
OBJ	An `object`/`JToken` member or a non-string dictionary key	Reads bind `JsonElement` instead of `JObject`; key conversion differs

B6 and B7 are checked only for the closure of request bodies and hub arguments. J4 and the non-public [JsonProperty] half of B8 scan every type in the SyRF assemblies the API loads, not only the wire closure.

The scan of main before R0 reported the 14 Newtonsoft-only attributes of plan §1.3 (J4) and the private StageSecuritySettings._customStagePermissions field (B8). PR-0.2 twinned 13 of them; the one J4 finding left is below. It also found more B1 members than the plan's grep: the ProjectDbBaseDto members of five project responses and AnnotationSessionDetailDto.Annotations, besides QuestionOptionDto.ParentFilter, TargetDto.ConditionalParentAnswers, SessionSubmissionWebDto.Annotations and NextStudyResponseDto.NextStudy. StatsWithIncompleteDto is not on the wire.

STJ attribute twins (house dual pattern)¶

Withheld: ProjectAgreementThreshold.AgreementMode. MassTransit already serialises with STJ, and the MassTransit golden shows IUpdateStudyScreeningStatsCommand carrying a ProjectAgreementThreshold with the computed agreementMode in its JSON. The STJ [JsonIgnore] twin would remove that field from the message. No consumer reads it (STJ uses the public parameterless constructor and the property has no setter), but plan 0.2.2 requires the backend-plan owner's agreement for any contract diff, and §12 sequences this type after backend plan PR-11 (C15, the agreement threshold lost in MassTransit deserialisation). The twin is a one-line follow-up once PR-11 lands or the owner agrees; the allowlist records it as fix by that follow-up.

Until R4 removes Newtonsoft, every Newtonsoft serialisation attribute carries its STJ twin beside it, never instead of it:

Newtonsoft	STJ twin
`[JsonIgnore]`	`[System.Text.Json.Serialization.JsonIgnore]`
`[JsonProperty(NullValueHandling = NullValueHandling.Ignore)]`	`[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]`
`[JsonProperty]` on a non-public member	`[JsonInclude]`
`[JsonProperty("name")]`	`[JsonPropertyName("name")]`
`[JsonConverter(typeof(StringEnumConverter))]`	`[JsonConverter(typeof(JsonStringEnumConverter))]`

Promotion safety: why the twins change no output today¶

STJ honours the twins wherever production code already uses STJ, so "attribute-only" holds only if no such path reaches a twinned type: an Entity/AggregateRoot subclass, StageSecuritySettings, ProjectReviewerStats or AllocationRegimeOrigin. [JsonInclude] and JsonStringEnumConverter would add output, which is why they matter most. A sweep of every production STJ call site in R0 found none that reaches them:

Automated. StjTwinReachability walks the closure STJ itself resolves. It honours [JsonIgnore] and [JsonInclude], and follows elements, keys and values. StjPromotionSafetyTests covers:
every site whose type is nameable (API, PM.Core, SharedKernel, the s3-notifier's shared value objects, the PDF agent's journals, proofs, scratch markers and health snapshot);
every MassTransit contract, which also covers the Quartz message scheduler and DelegatedWorkGate. Only IUpdateStudyScreeningStatsCommand reaches a candidate: the withheld ProjectAgreementThreshold.
By inspection. These sites carry no twinned type:
Anonymous objects of primitives: Program.cs:803, the IdentityService/AccountController request bodies, UserImpersonationMiddleware, the bundle-reader budgets, the s3-notifier log payloads.
JsonElement reads in BffAuthController and AccountController.
Dictionary<string, object> holding strings, numbers and booleans (the BFF PKCE cookie, the health response).
The Identity host and migration tool, which reference only SharedKernel and Identity.Shared and name no Entity/AggregateRoot type.
Pinned by goldens. swagger.json generation (0.2.1), MassTransit contracts (0.2.2) and the checkpoint definition serialisers (J8, 0.1.6).
Not STJ. MVC and the hub (Newtonsoft today), the Quartz JobDataMap (Newtonsoft) and the s3-notifier DynamoDB outbox (attribute values).

Persisted bytes the migration must not move by accident¶

FEAT-024 submission digests (0.1.5, D6). ProjectStatisticsSubmissionDigest hashes Newtonsoft JToken.FromObject output and the digest is persisted. Absolute pins cover each screening decision with and without reconciliation, each of the eight Annotation subtypes, IsDirty with pending Events, null notes, parent and answer, a decimal with a trailing zero, an OutcomeData with fractional time points and ExtraElements, each SessionStatus and an explicit entity order. The post-binding pins bind synthetic JSON through today's MVC settings and map it as ReviewController.SubmitSession does. A digest change is allowed only at a ship point: the PR re-commits the values and records that staging statistics are reset and production holds no stored digests (plan D6).
Decimal binding today. Through MVC's default FloatParseHandling.Double, AnnotationConverter reads decimal answers through a double: 1.50 binds as 1.5, 0.30000000000000004 as 0.3 and 12345678901234567.89 as 12345678901234600. TodaysBindingReadsDecimalsThroughADouble records this for R1 (1.1.4).
Definition metadata (0.1.6). StageStatisticsDefinitionSnapshot.Serialize, ReviewerAnnotationScopeCalculator.DescribeDefinition, QuestionAnswersScopeCalculator.DescribeDefinition and ReviewerProgressDefinition.Serialize use STJ default options and feed stored checkpoint digests. They stay on default options; the allowlist's defaultOptionsSerializers section records them and each has a golden.
JsonConvert.DefaultSettings (0.1.7) stays unset. A source scan fails on any assignment under src; the real API host, the PM container and the Quartz host assert it is null at runtime.

Known differences and observations¶

#	Observation	Status
B12	The statistics admin 409 writes `"status":409` and then the extension `"status":"<lifecycle>"`, so `JSON.parse` keeps the extension	Pinned byte-for-byte (0.1.4); keep it through the migration (D10)
—	Newtonsoft's `ProblemDetailsConverter` writes `Extensions` keys as given, not camelCased, while objects inside an extension are camelCased	Pinned by `rest/ProblemDetails.golden.json`
B14	Newtonsoft writes whole doubles as `1.0`; STJ writes `1`	Intentional difference (the comparator normalises numbers)
D5	JSON Patch bodies stay on a Newtonsoft formatter until R4	Allowlisted (`Operation.value`)

Flag decision¶

R0 is not flagged: it changes no runtime behaviour. The goldens, scanner, pins and guards are tests, and the STJ twins are read only by STJ, which serves no API traffic until the R1/R2 switches (stjMvcSerializer, stjHubProtocol) exist.