Replace the shared `rabbit` administrator with least-privilege broker principals¶

This is the implementation plan for the credential half of the broker security contract (B0). It extends the contract: the target model, the naming, the transport design and the B2 end state stay as written there. This plan updates the facts to 2026-10-14, closes the gaps the contract left open, and orders the work into reversible phases. Chris approved writing it on 2026-10-14 and accepted every decision in §11 as recommended on 2026-10-14.

This document changes nothing live. Phase 0, Phase 1 (steps 1 to 4), Phase 2 steps 1 and 2, D11 and Phase 6 step 1 are done; Phase 2 step 3's chart switch (syrf#4310) moves each preview to syrf-pr-N as it redeploys from a head that includes it (§1); every remaining phase needs its own PR and its own approval. Each step marked (P) touches production or the broker that production shares, and needs Chris's explicit go for that step.

Contents¶

Brief versus reality
Evidence and markers
Hard operational rule
1. Where we are on 2026-10-15
2. Target model
3. Permission regexes per principal
4. How the broker learns the principals
5. Secret distribution
6. Management UI exposure
7. Phases
8. Production impact per phase
9. Acceptance criteria
10. Wolverine and the MassTransit 8.5.11 bridge
11. Decisions for Chris
12. Superseded and related PRs
13. Risks
14. Follow-ups outside this plan

Brief versus reality¶

The brief for this plan stated some facts that git and the cluster contradict. The plan follows the evidence:

#	Brief	Evidence	Effect on the plan
R1	MassTransit 8.5.11 today	Every `MassTransit` package on `main` is 8.4.0* (V). The 8.5.11 bridge is open as syrf#4147 (ADR-021 P1).	Regexes are drafted against 8.4.0. The fixture must re-run on #4147's head before either merges after the other (§10). Resolved: #4147 merged first, and Phase 0.1–0.2 (syrf#4201) proved the regexes on 8.5.11.
R2	One GCP Secret Manager secret per principal	The five principals that already exist get their passwords from an in-cluster ESO `Password` ClusterGenerator (`CreatedOnce`), never stored in GCP SM (cluster-gitops#1430). The broker's Erlang cookie uses the same generator (#1633).	The plan keeps the generator and uses GCP SM only for the break-glass administrator. Decision D3.
R3	The broker learns users from the ESO definitions file	True, but RabbitMQ reads that file only at boot (#1630). Every new principal or permission change would need a restart of the shared broker, which is a 2 to 5 minute outage for production.	The plan adds a live definitions applier, so phases 1 to 4 restart nothing. Decision D1.
R4	Consumers are API, PM, Quartz, previews and notifiers	A Kubernetes PDF agent also runs in `syrf-staging` (`pdf-agent-staging-syrf-pdf-agent`, `ConsumerState=Enabled`) as `rabbit` (V). `.claude/rules/pdf-agent.md` says no PDF agent runs in Kubernetes.	It gets its own principal. Decision D6.
R5	The production notifier's ACK `Function` CR holds the password in plain text	All eight ACK `Function` CRs do: production, staging (notifier and reconciler) and five previews. Each has a `RabbitMqPassword` key in `spec.environment.variables` (V, key names only).	§5 covers all of them.
R6	(not in the brief)	GitHub repository secrets `RABBITMQ_PASSWORD` exist in `camaradesuk/syrf` (2025-12-11, referenced by no workflow) and `camaradesuk/camarades-infrastructure` (2026-05-04, feeds the dormant Terraform preview-Lambda module). The local `rabbitmq` MCP server logs in as `rabbit` with `--allow-mutative-tools`.	Phase 5 removes them.
R7	(not in the brief)	Deleting the management Ingress would leave `rabbitmq.camarades.net` with no DNS owner. External-DNS runs `policy: upsert-only` and `registry: noop` and has no other source for that name. AMQPS on 5671 (Lambdas, ARRNC) depends on that name.	§6 gives the record an owner; #1639 removed the Ingress first.

Evidence and markers¶

The markers are the contract's:

Marker	Meaning
V	Verified on 2026-10-14 by read-only `kubectl get` (key and field names only, never Secret values) on `gke_camarades-net_europe-west2-a_camaradesuk`, or read from `camaradesuk/cluster-gitops` `origin/main` `c04bdecc`, `camaradesuk/camarades-infrastructure` `origin/main` or `camaradesuk/server-config` `origin/main`.
S	Derived from SyRF source on `main` `19c5797af` (MassTransit 8.4.0).
U	Unknown. It blocks the step that depends on it. It never means "absent".

The broker was not queried for this plan, and is not queried to verify it. No rabbitmqctl, no management API, no MCP broker tool and no port-forward was used. Broker-side facts come from merged PR bodies, the definitions applier's logged results and pod logs.

Hard operational rule¶

Never run looped rabbitmqctl, or an unfiltered queue, exchange, binding, channel or connection listing, against the shared broker. On 2026-10-10 that pattern OOM-killed rabbitmq-0, which took production messaging down.

It applies to people, agents, CI, the definitions applier and the preview reconciler alike:

Chris's standing rule: never query the shared broker directly. No rabbitmqctl, no management API call and no MCP broker tool, from anyone or anything, unless Chris gives an explicit go for that one call. Verify with kubectl and logs instead.
Default evidence for every phase:
The applier's logged result. The definitions applier logs the HTTP status and the sha256 of each import. An import that fails partway returns 4xx (#4200 proved it), so a 204 means every entry in the file was applied.
ExternalSecret status (SecretSynced) and the applier Job and CronJob status.
Argo health, pod readiness, the API/PM/Quartz/PDF-agent health endpoints, MassTransit "bus started" and reconnect log lines, workload logs for auth or refusal errors (ACCESS_REFUSED, Broker unreachable), rabbitmq-0 start time and restart count, and CloudWatch logs for the Lambdas.
Direct broker calls, only with Chris's explicit go for that call: one filtered call at a time, scoped to a vhost or a single object, with columns= and disable_stats=true, made as syrf-broker-provisioner (an administrator). The syrf-broker-observer principal (tag monitoring) cannot do user or permission reads: it gets 401 on every /api/users* and /api/permissions* endpoint, because those are administrator-only in RabbitMQ 3.13 (proved locally in cluster-gitops#1664). It can read a vhost's connections (GET /api/vhosts/<vhost>/connections?columns=user,ssl) only if it holds a ^$ ^$ ^$ permission entry on that vhost, which it does not today. Never call /api/queues, /api/exchanges, /api/bindings, /api/channels or /api/connections without a vhost (and, for queues, exchanges and bindings, an object name).
The only cluster-wide listings ever allowed are the small name-only ones (GET /api/vhosts and GET /api/users, with columns=name, plus tags for users, and disable_stats=true). The preview reconciler uses these as the provisioner; people and agents need Chris's go first.
No kubectl exec … rabbitmqctl in automation. Holds since 2026-10-17 (AC12): syrf#4251 removed the _preview-cleanup.yml vhost deletion and syrf#4274 removed preview-sweep.yml "Layer 4a"; the preview reconciler's teardown replaces both.
No live test messages. Only organic traffic is observed, as the contract requires.
Credential switches re-sync the notifiers in the same change (hard rule). Every later change of a credential or principal that a notifier uses (Phases 3 to 5, including the Phase 5.3 rabbit rotation) must re-sync each affected environment's notifier env-vars (s3-notifier-<env> and its reconciler) in the same change, and verify the Lambda's configuration before the old credential is removed. This is risk K3 and decision D9. Cost of forgetting it: see the 2026-10-14 incident in §1.

1. Where we are on 2026-10-15¶

Phase 0 and Phase 1 are complete. D11 and Phase 6 step 1 are done. D13 is moot. Nothing uses the new principals yet: their consumer Secrets exist in syrf-staging and syrf-production, but no workload reads them.

Done (V)¶

Contract step	What	PR
B1.3	AMQPS listener on 5671 (Let's Encrypt `rabbitmq-amqps-tls`, server-authenticated); the broker loads the ESO-rendered `rabbitmq-definitions` at boot; memory 1Gi, watermark 0.6	cluster-gitops#1431
B1.4	ingress-nginx L4 passthrough for 5671	#1432
B1.6, part of B1.7 and B1.10	Principals `syrf-staging-s3notifier`, `syrf-production-s3notifier` (`.` on their vhost, tag none), `syrf-staging-pdfagent` (Paused set `^$ ^$ ^$`), `syrf-production-pdfagent` (`.` on `syrf-production`), `syrf-broker-observer` (tag `monitoring`, no permissions). Consumer Secret `rabbitmq-s3notifier` in `syrf-staging` and `syrf-production`.	#1430, #1630
B1.5, B1.5b	The broker uses its own generated Erlang cookie in `rabbitmq` only. `rabbit-mq` copies hold only `rabbitmq-password`. NetworkPolicy narrowed (declared intent: the cluster does not enforce NetworkPolicy).	#1632–#1635
B1.0 (code)	Strict TLS validation for 5671 in every SyRF bus (`RabbitMqTls`)	syrf#3878
Phase 0	Fixtures, E2E, local proofs, parity check, API chart switch: see Phase 0	syrf#4201, syrf#4225, syrf#4200, cluster-gitops#1656, syrf#4199
Phase 1 steps 1 and 2, step 3 first run	Eight `CreatedOnce` ExternalSecrets (staging and production `api`, `pm`, `quartz`, `syrf-staging-pdfagent-gke`, `syrf-broker-provisioner`), all `SecretSynced`. The seven workload principals have interim `.*` on their own vhost; the provisioner is `administrator` with no permissions. The live applier (CronJob every 10 minutes plus a PostSync Job) made its first import as `rabbit` at 2026-10-15T01:46:27Z: HTTP 204, sha256 `8cd85bd7…` recorded. Merged 2026-10-15T01:45Z.	cluster-gitops#1664
Phase 1 step 3 switch	The applier authenticates as `syrf-broker-provisioner` (`APPLIER_IDENTITY=provisioner`). Merged 2026-10-15T03:21Z. No import has run since (the definitions sha256 is still `8cd85bd7…`), so the next definitions change is the provisioner's first import.	cluster-gitops#1668
Phase 1 step 4	ClusterSecretStores `rabbitmq-syrf-staging` and `rabbitmq-syrf-production` (kubernetes provider, `conditions.namespaces` set to that environment, reader ServiceAccount whose Role may `get` only that environment's principal Secrets). Consumer ExternalSecrets `rabbitmq-api`, `rabbitmq-pm`, `rabbitmq-quartz` in both namespaces and `rabbitmq-pdfagent-gke` in `syrf-staging`, key `rabbitmq-password`. Verified 2026-10-17: both stores `Valid`, all seven `SecretSynced`, each Secret holds only `rabbitmq-password` (48 characters); `rabbitmq-0` and the environment pods did not restart. Staging merged 2026-10-17, production 2026-10-17T16:51Z.	#1688, #1690
D11	`syrf-production-pdfagent` set to `^$ ^$ ^$`; live with the 01:46Z import. No `syrf-pdf-agent-production` container exists on arrnc-api (confirmed 2026-10-15).	cluster-gitops#1663
Phase 6 step 1	`external-dns.alpha.kubernetes.io/hostname: rabbitmq.camarades.net` on Service `ingress-nginx/ingress-nginx-controller`. Verified live: the annotation is present, the IP is still `34.13.63.21`, and `openssl s_client -connect rabbitmq.camarades.net:5671` gives verify 0. The record is owned again.	cluster-gitops#1662
Bus-endpoint names	Corrected in the committed files, the server-config scope and the parity check	syrf#4228, server-config#306, cluster-gitops#1659

Phase 1 evidence (no broker query). rabbitmq-0 started 2026-10-14T04:03:34Z and has 0 restarts, so the import restarted nothing (AC11 holds so far). No workload logged a broker authentication or connection error after the import.

Phase 1 is complete (2026-10-17). The applier's switch to syrf-broker-provisioner (cluster-gitops#1668) is live but has not yet imported; the first definitions change after it proves the provisioner's import. Step 4 is #1688 and #1690.

Incident: production notifications lost, 2026-10-14 (D13)¶

cluster-gitops#1437 and #1438 (the B1 rabbit password rotation) were merged by another session around 04:00Z on 2026-10-14, not closed as this plan's D13 intended. The production s3-notifier Lambda only received the new password at 09:43Z, so every production upload notification between about 04:00 and 09:43 failed with "Broker unreachable". One search import was lost (project dc7067e2…, search ec464edc…, 08:35Z); Chris chose not to replay it. D13 is therefore moot, and this plan's §12 warning about exactly this failure came true. The lesson is now a hard rule: every credential switch re-syncs each environment's notifier env-vars in the same change (Hard operational rule, K3, D9). Phase 5.3 must still rotate rabbit again, under that rule.

Still `rabbit` (V)¶

In-cluster: syrf-api, syrf-projectmanagement, syrf-quartz in syrf-production, syrf-staging and five previews (pr-2412, pr-3292, pr-3327, pr-3394, pr-3934), plus the staging Kubernetes PDF agent. All dial amqp://rabbitmq.rabbitmq.svc.cluster.local:5672 and read rabbit-mq key rabbitmq-password.
Lambdas: all eight notifier and reconciler functions, over public plaintext 5672. The switches are open as (status as of 2026-10-14; re-check before use) #1433 (staging), #1434 (production) and #1435 (previews, still as rabbit).
Admin paths: the API chart's preview PreSync hook (presync-rabbitmq-vhost.yaml) creates pr-N vhosts through the management API as rabbit in previews whose head predates syrf#4310 (Phase 2 step 3). The public management Ingress rabbitmq.camarades.net (443) was removed on 2026-10-14 by cluster-gitops#1639; the DNS name was kept (External-DNS upsert-only) and got an owning source again on 2026-10-15 through cluster-gitops#1662 (Phase 6 step 1). The local rabbitmq MCP server uses rabbit, and since #1639 its public URL no longer reaches the API.
Definitions: rabbit keeps tag administrator and .* on /, syrf-production, syrf-staging and local. Its password was rotated on 2026-10-14 by #1437/#1438 (see the incident above).

2. Target model¶

This matches the contract's §2 with three additions: the provisioner (D1), the staging Kubernetes PDF agent (D6) and preview teardown.

Principal	Vhost	Tags	Used by	Status
`syrf-production-api`, `-pm`, `-quartz`	`syrf-production`	none	the three Deployments	new
`syrf-production-s3notifier`	`syrf-production`	none	`syrfAppUploadS3Notifier`	exists, `.*`
`syrf-production-pdfagent`	`syrf-production`	none	ARRNC production slot (not cut over)	exists, `.*`; set to the Paused set (D11)
`syrf-staging-api`, `-pm`, `-quartz`	`syrf-staging`	none	the three Deployments	new
`syrf-staging-s3notifier`	`syrf-staging`	none	staging notifier and reconciler Lambdas	exists, `.*`
`syrf-staging-pdfagent`	`syrf-staging`	none	ARRNC staging slot (Paused)	exists, Paused set
`syrf-staging-pdfagent-gke`	`syrf-staging`	none	the Kubernetes staging PDF agent until it retires	new (D6)
`syrf-pr-N`	`pr-N` only	none	that preview's API, PM, Quartz and Lambda	new (D2)
`syrf-broker-observer`	none	`monitoring`	metrics and, if it is given a `^$ ^$ ^$` entry on a vhost, that vhost's connections. It gets 401 on `/api/users` and `/api/permissions` (administrator-only in 3.13). Direct calls need Chris's go	exists
`syrf-broker-provisioner`	none	`administrator`	the definitions applier and the preview reconciler, `rabbitmq` namespace only	new (D1); this is the contract's standing admin identity (named `syrf-topology-operator` there)
`syrf-broker-breakglass`	none	`administrator`	dormant; Chris approves each use	new (contract B2)

Rules carried from the contract:

No workload principal has a tag, so none can use the management UI or API.
No principal has permissions in more than one vhost. Vhosts stay as a second boundary, never the only one.
Exactly two administrators remain at the end: the provisioner (standing, automation only) and the break-glass user (dormant). Both credentials live only in the rabbitmq namespace (and, for break-glass, GCP Secret Manager). No workload, Lambda or ARRNC host holds an administrator credential.
Rotation creates a second principal (-api-2), switches to it, then deletes the old one. A live password is never edited in place, except by the preview reconciler's repair upsert, which re-applies the same password from the preview's Secret (§5).

3. Permission regexes per principal¶

Method¶

The PDF agent already sets the pattern: src/services/pdf-agent/broker-permissions.json is the contract, and PdfAgentBrokerPermissionTests runs the agent's real MassTransit registration against a real broker as a user holding exactly those regexes. Any ACCESS_REFUSED fails the test. This plan applies the same method to every workload:

Commit broker-permissions.json next to each service (api, project-management, quartz, s3-notifier), with the same {busEndpointPrefix} placeholder convention.
A broker-permission test per service runs its production registration (ConfigureSyrfMassTransit with the real consumers and definitions) under those regexes and exercises every path: start-up declaration, each publish, send, request, schedule, job submission, fault and scheduled redelivery. Negative tests are listed in the contract's §3 and below.
The cluster-gitops definitions carry these strings verbatim. A check compares them with the syrf file at the deployed image's commit (Phase 0.5).

The regexes below are drafts derived from source (S). As for the PDF agent, nothing goes live until the fixture proves each set. Names the fixture must pin are marked U.

Names in use on `main` (S)¶

These extend the contract's §3 with what has changed since 65ae1099b:

Default-formatted PM endpoints (no SyRF. prefix): now eight, not five. StartDailyProjectStatistics, ObserveDailyProjectStatistics, RunProjectStatisticsMaintenance, RunProjectStatisticsDriftCheck, RunProjectStatisticsRepair, plus FoldProjectStatistics, RunProjectStatisticsFoldSweep and ProjectStatisticsFoldStampAdvance (their definitions set only ConcurrentMessageLimit).
New PM command queues: the seven notifier-authority queues (SyRF.ProjectManagement.I{GetBulkPdfReleaseState,ResolveBulkPdfStorageBinding,RegisterBulkPdfCleanup,RegisterBulkPdfQuarantineFence,TransferBulkPdfQuarantineFence,AcknowledgeBulkPdfCleanup,ConfirmBulkPdfHoldRetired}Command), IReconcileBulkPdfReceiptDrainCommand, IRunBulkStudyUpdateOperationCommand (a job PM submits to itself) and IBulkPdfUploadObjectReadyCommand.
Recurring schedules (PM → Quartz): StartDailyProjectStatistics, RunProjectStatisticsDriftCheck, RunProjectStatisticsRepair, RunProjectStatisticsMaintenance, RunProjectStatisticsFoldSweep. One-shot schedules: IObserveDailyProjectStatisticsCommand, IMarkSessionIdleCommand, IRemoveIdleSessionCommand, IRemoveSuspendedSessionCommand, ICheckConnectionLivenessCommand and the saga timeout ISearchUploadTimeoutExpiredEvent. A state-machine Schedule sends to the saga's own input address, so its destination is SyRF.ProjectManagement.SearchImportJobState, not an event exchange.
Scheduled redelivery (Quartz sends back to the input queue): the BulkPdfUpload endpoint, IBulkPdfUploadObjectReadyCommand, the four review-session queues (IMarkSessionIdle, IRemoveIdleSession, IRemoveSuspendedSession, ICheckConnectionLiveness) and the PDF agent's IProcessBulkPdfUploadCommand.
Bus endpoints. MassTransit 8.5.11 names a bus endpoint <host>_<process>_bus_<id> (DefaultEndpointNameFormatter.GetTemporaryQueueName): <host> is Environment.MachineName (the pod name in Kubernetes), <id> is a 26-character lower-case NewId, and every non-alphanumeric is removed from both parts. Every image runs dotnet <dll>, but BusHostInfo replaces a process name of dotnet with the entry assembly's file name, so <process> is SyRFAPIEndpoint, SyRFProjectManagementEndpoint, SyRFQuartz or SyRFPdfAgent, never dotnet. When host + process + 3 + 29 would exceed 72 characters, the formatter cuts the host to 14 characters, then, if still over, the process to 14. Proved 2026-10-15 on a local RabbitMQ 3.13.6, with the real agent image syrf-pdf-agent:1.123.0-sha.960df11 and with dotnet <dll> hosts carrying the services' assembly names under live pod names:

Workload	Host name	Bus endpoint
API	`syrf-api-59c985b8d-fz99q`	`syrfapi59c985b8dfz99q_SyRFAPIEndpoint_bus_<id>`
PM	`syrf-projectmanagement-7d998b7857-nzrjr`	`syrfprojectman_SyRFProjectMan_bus_<id>` (both cut, so every PM pod has this prefix)
Quartz	`syrf-quartz-75bcb5997d-s964l`	`syrfquartz75bcb5997ds964l_SyRFQuartz_bus_<id>`
PDF agent, ARRNC	`syrf-pdf-agent` (pinned)	`syrfpdfagent_SyRFPdfAgent_bus_<id>`
PDF agent, Kubernetes	`pdf-agent-staging-syrf-pdf-agent-d5f96c46c-zhnp2`	`pdfagentstagin_SyRFPdfAgent_bus_<id>` (host cut)

With a pod-template hash of at most 10 characters, the stripped API host is at most 22 characters (the cut starts above 25) and the Quartz host at most 25 (above 30), so neither is ever cut. The Lambda's machine and process names are U. scripts/broker-principals.py (#4225) computes the same prefixes. - #4139/#4140 add PM's ingress exchange SyRF.ProjectManagement.BulkPdfUpload.Agent and move the agent to it. The PM regex below already allows it. The agent's file changes in #4140.

Shared fragments¶

Name	Regex fragment
`PMQ`	`SyRF\.ProjectManagement\.[A-Za-z]+(_error\\|_skipped\\|\.Agent)?`
`PMDEF`	`(StartDailyProjectStatistics\\|ObserveDailyProjectStatistics\\|RunProjectStatistics(Maintenance\\|DriftCheck\\|Repair\\|FoldSweep)\\|FoldProjectStatistics\\|ProjectStatisticsFoldStampAdvance)(_error\\|_skipped)?`
`MSG`	`SyRF\.(ProjectManagement\.Messages\.(Commands\\|Events)\\|API\.Messages\.Events\\|S3FileSavedNotifier\.Messages\.Events\\|SharedKernel\.Interfaces\\|ProjectManagement\.Endpoint\.Sagas):[A-Za-z]+`
`SIGNALR`	`SyRF\.API\.SignalR\.(Statistics:ProjectStatisticsChanged\\|ClaimRevocations:ActivityClaimRevoked)`
`APITMP`	`api-(project-statistics-invalidations\\|activity-claim-revocations)-?[0-9a-f]{32}` (separator U)
`SCHED`	`MassTransit\.Scheduling:[A-Za-z]+`
`FAULT`	`MassTransit:Fault(--.+--)?`
`JOBS`	`MassTransit\.Contracts\.JobService:.+`
`QTZ`	`(quartz\\|Job\\|JobType\\|JobAttempt)(_error\\|_skipped)?`
`BUS_API` / `BUS_PM` / `BUS_QTZ`	`syrfapi[a-z0-9]+_SyRFAPIEndpoint_bus_[a-z0-9]+` / `syrfprojectman_SyRFProjectMan_bus_[a-z0-9]+` / `syrfquartz[a-z0-9]+_SyRFQuartz_bus_[a-z0-9]+`
`BUS_AGENT`	the PDF agent's pinned `{busEndpointPrefix}_[a-z0-9]+` for each slot that talks to this vhost
`API_TO_PM`	`SyRF\.ProjectManagement\.(IUpdateStudyScreeningStatsCommand\\|IGetBulkPdfReleaseStateCommand\\|IResolveBulkPdfStorageBindingCommand\\|IRegisterBulkPdfCleanupCommand\\|IRegisterBulkPdfQuarantineFenceCommand\\|ITransferBulkPdfQuarantineFenceCommand\\|IAcknowledgeBulkPdfCleanupCommand\\|IConfirmBulkPdfHoldRetiredCommand\\|IStartSearchUpdateCommand)`
`QTZ_DEST`	SyRF\.ProjectManagement\.Messages\.Commands:I(MarkSessionIdle\\|RemoveIdleSession\\|RemoveSuspendedSession\\|CheckConnectionLiveness\\|ObserveDailyProjectStatistics\\|StartDailyProjectStatistics\\|RunProjectStatistics(Maintenance\\|DriftCheck\\|Repair\\|FoldSweep))Command\\|SyRF\.SharedKernel\.Interfaces:ICommand\\|SyRF\.ProjectManagement\.(SearchImportJobState\\|BulkPdfUpload\\|I(BulkPdfUploadObjectReady\\|MarkSessionIdle\\|RemoveIdleSession\\|RemoveSuspendedSession\\|CheckConnectionLiveness\\|ProcessBulkPdfUpload\\|StartParsingReferenceFile\\|StartRobProcessingJob\\|StartBulkStudyUpdateJob\\|RunBulkStudyUpdateOperation)Command)

(\| is a table escape. The committed JSON uses a bare |.)

Draft sets¶

Every set is wrapped as ^(…)$.

Principal	configure	write	read
`-pm`	`PMQ`, `PMDEF`, `BUS_PM`, `MSG`, `SCHED`, `FAULT`, `JOBS`, plus `(Job\\|JobType\\|JobAttempt)` if job-consumer status sends declare them (U)	configure set, plus `BUS_API`, `BUS_AGENT` (responses)	`PMQ`, `PMDEF`, `BUS_PM`, `MSG`, `FAULT`, `JOBS`
`-api`	`BUS_API`, `APITMP`, `SIGNALR`, `SyRF\.API\.Messages\.Events:[A-Za-z]+`, `SyRF\.ProjectManagement\.Messages\.Commands:IFoldProjectStatisticsCommand`, `SyRF\.SharedKernel\.Interfaces:[A-Za-z]+`, `API_TO_PM`, `SCHED`, `FAULT`, `JOBS`	same as configure	`BUS_API`, `APITMP`, `SIGNALR`, `SyRF\.API\.Messages\.Events:[A-Za-z]+`, `SyRF\.ProjectManagement\.Messages\.Commands:IFoldProjectStatisticsCommand`, `API_TO_PM` (see D14), `FAULT`, `JOBS`
`-quartz`	`QTZ`, `BUS_QTZ`, `SCHED`, `JOBS`, `FAULT`, `QTZ_DEST`	configure set, plus `BUS_API`, `BUS_PM`	`QTZ`, `BUS_QTZ`, `SCHED`, `JOBS`, `FAULT`
`-s3notifier`	`SyRF\.ProjectManagement\.IBulkPdfUploadObjectReadyCommand`, `SyRF\.S3FileSavedNotifier\.Messages\.Events:ISearchUploadSavedToS3Event`, `SyRF\.SharedKernel\.Interfaces:(ICanStartSearchImportJob\\|IIntegrationEvent)`, `MassTransit\.Contracts\.JobService:SubmitJob--SyRF\.ProjectManagement\.Messages\.Commands:IStartBulkStudyUpdateJobCommand--`, its bus endpoint if one is created (U)	same as configure	`SyRF\.ProjectManagement\.IBulkPdfUploadObjectReadyCommand` (D14), `SyRF\.S3FileSavedNotifier\.Messages\.Events:ISearchUploadSavedToS3Event`, the `SubmitJob` exchange above
`-pdfagent`, `-pdfagent-gke`	`broker-permissions.json`: `Paused` or `Enabled` per slot state (with #4140's ingress exchange). The ARRNC agent's prefix is `syrfpdfagent_SyRFPdfAgent_bus` (pinned host `syrf-pdf-agent`). The Kubernetes agent's pod name is cut to 14 characters, so its prefix is the fixed `pdfagentstagin_SyRFPdfAgent_bus` and its pattern `pdfagentstagin_SyRFPdfAgent_bus_[a-z0-9]+`.
`syrf-pr-N`	`.*`	`.*`	`.*` (on `pr-N` only; previews hold no data that warrants finer grants, and one principal serves four workloads)

Negative tests the fixture must add to the contract's list:

-api cannot consume, purge or delete SyRF.ProjectManagement.SearchImportJobState or any default-formatted statistics queue.
-quartz cannot publish to any SyRF.API.* exchange or to a PM command it does not schedule (for example IStartParsingReferenceFileCommand via its message-type exchange).
-s3notifier cannot publish IProcessBulkPdfUploadCommand or consume any queue.
No syrf-staging-* principal can open syrf-production, and syrf-pr-N cannot open pr-M or syrf-staging.
Every principal is refused queue.declare on quartz unless it is -quartz.

Known residuals¶

Configure implies delete (contract D6). It applies to -api and -s3notifier on their queue: destinations, to -quartz on every redelivery queue, and to -pm on its own queues.
Read on a queue: destination's exchange is also read on the same-named queue. RabbitMQ does not distinguish resource kinds, so -api and -s3notifier could consume or purge the PM queues they send to. The fix is the #4138 pattern: PM-owned ingress exchanges with different names, and exchange: sends. Decision D14.
-pm reaches the PDF agent's work queue. The release gate's queue: send to SyRF.ProjectManagement.IProcessBulkPdfUploadCommand needs configure on that name and read on the same-named exchange, so -pm can also consume, purge or delete the agent's queue. The agent's _error and _skipped queues are not granted (syrf#4201 narrowed PMQ's dead-letter alternative to names that do not start with IP). The same D14 follow-up closes it: an agent-owned ingress exchange and an exchange: send.
-pm can delete other services' bus-endpoint exchanges. A response declares the requester's bus-endpoint exchange, so -pm needs configure on the API's and the PDF agent's bus endpoints, which also allows exchange.delete. That drops the binding to the pod's bus-endpoint queue, and responses to that pod are lost until it reconnects. The queue itself is exclusive to its owner's connection (delete, purge and consume answer 405 RESOURCE_LOCKED), and -pm holds no read on it.
All three -pm residuals are listed in src/services/project-management/broker-permissions.json (residuals) and pinned by ProjectManagementBrokerPermissionTests.KnownResiduals_TheAgentsWorkQueueAndOtherServicesBusEndpointExchanges.
Quartz is a confused deputy (contract §Risks). Anyone who can write MassTransit.Scheduling:* can have Quartz deliver to anything in QTZ_DEST. That is why QTZ_DEST is an explicit allowlist.

Phase 0.1–0.2 results (proved)¶

syrf#4201 committed broker-permissions.json for api, project-management, quartz and s3-notifier and a broker-permission test per service, on MassTransit 8.5.11. Each test runs the service's production registration as a user holding only the committed strings, and its counterpart buses take their own bus-endpoint names, so a grant on one service's bus endpoint cannot cover another's. The files are the source of truth; the drafts above are kept for the derivation. The fixture changed the drafts as follows (S, proved):

Principal	Change from the draft	Why
`-api`	`APITMP` is `api-(project-statistics-invalidations\\|activity-claim-revocations)[0-9a-f]{32}(_error)?`	MassTransit appends the 32-hex `InstanceId` with no separator, and a faulted fan-out delivery creates `<queue>_error`
`-api`	`IStartSearchUpdateCommand` removed from `API_TO_PM`	`StudyController` injects its request client but never sends
`-pm`	`MSG` ends `[A-Za-z0-9]+`	`SyRF.S3FileSavedNotifier.Messages.Events:ISearchUploadSavedToS3Event` has a digit; the saga binds it at start-up
`-pm`, `-quartz`	add `Instance[a-z0-9]+` (PM: configure, write, read; Quartz: configure, write)	The 8.5 job service gives each job consumer host a per-instance endpoint `Instance<NewId>`; the `JobAttempt` saga sends `StartJob` there
`-pm`	`BUS_API` and `BUS_AGENT` move into configure as well as write	A responder declares the requester's bus-endpoint exchange (auto-delete) before publishing to it; write alone refuses every response. New residual, see Known residuals
`-pm`	`PMQ`'s `_error`/`_skipped` alternative is `SyRF\.ProjectManagement\.(I[A-OQ-Z]\\|[A-HJ-Z])[A-Za-z]*(_error\\|_skipped)`	Leaves out the PDF agent's `IProcessBulkPdfUploadCommand_error`/`_skipped` without a lookahead; the start-up test checks every PM endpoint's dead-letter names stay covered
`-quartz`	write on `BUS_API` and `BUS_PM` removed	Quartz sends nothing to either bus endpoint (proved under distinct names), and a response would need configure anyway
`-pm`, `-quartz`	`RunDeletionLifecycle` added to `PMDEF` (PM) and `IRunDeletionLifecycleCommand` to `QTZ_DEST` (Quartz)	Deletion lifecycle S3a (main `e0a647712`) adds a default-formatted runner endpoint and its recurring schedule; both were refused under the earlier sets
`-api`	`APITMP` adds `search-deletions`; configure and read on `SyRF\.ProjectManagement\.Messages\.Events:ISearchDeletedEvent`	Deletion lifecycle S4a (syrf#4241): the API consumes project-management's `SearchDeleted` event on a third per-pod fan-out queue and never publishes it. PM's existing `MSG` grant covers the publish
`-s3notifier`	unchanged; no bus-endpoint grant	The Lambdas make no request, so their bus never creates a bus endpoint

The plan's negative list says -s3notifier cannot consume any queue. Under the accepted D14 residual it can consume SyRF.ProjectManagement.IBulkPdfUploadObjectReadyCommand (its send binds that name); the notifier test pins this residual and refuses every other queue. Likewise -api can consume the API_TO_PM queues and nothing else of project management's.

4. How the broker learns the principals¶

Today: boot-only definitions¶

Since #1431, RabbitMQ imports rabbitmq-definitions (rendered by ESO from the generated passwords) at boot only. The import creates or updates the listed users and permissions and never deletes anything. It also never revokes a grant whose entry is removed from the file (Phase 0.4, C1 and C11). A change in git therefore reaches the broker only at the next restart of rabbitmq-0. That restart is a 2 to 5 minute AMQP outage for every environment, production included (#1634 measured about 2 minutes at 18 vhosts).

Proposed: a live definitions applier (decision D1)¶

A small Argo-managed workload in the rabbitmq namespace imports the same rendered file live, with one POST /api/definitions per run. RabbitMQ applies definitions imported over HTTP immediately and upserts in the same way as the boot import. No restart is needed.

Form: a CronJob (every 10 minutes, concurrencyPolicy: Forbid, one request, 30 s timeout) plus the same Job as an Argo PostSync hook, so a merged change lands within minutes. The file is mounted from rabbitmq-definitions, and the Job skips the call when the file's hash equals the last applied hash (kept in a ConfigMap it owns).
Identity: at first it authenticates as rabbit, using the rabbitmq-namespace copy of rabbit-mq that the definitions template already reads. Its first import created syrf-broker-provisioner (2026-10-15T01:46:27Z, cluster-gitops#1664), and the next commit (#1668, merged 2026-10-15T03:21Z) switched the applier to the provisioner, so rabbit is no longer needed for provisioning.
Cost on the broker: one definitions import of a few dozen users and permissions, with no queues, exchanges or bindings in the file. This is far below the 2026-10-10 failure pattern. It never lists queues.
Result logging. Each run logs the HTTP status and the sha256 of the imported file. A 204 means every entry applied, because a partial failure returns 4xx (#4200); this log line is the default verification evidence for every phase (Hard operational rule).
Boot import stays. If the PVC is lost, the boot import rebuilds the same state.
Local proof (done 2026-10-14, Phase 0.4 proofs): on RabbitMQ 3.13.6, POST /api/definitions with plaintext password fields creates users, updates passwords and tags of existing users, and changes permissions without a restart (confirmed). Removing an entry deletes nothing (confirmed). Changing rabbit's tags to "" demotes it live for HTTP (confirmed); tags do not affect AMQP. Two findings change the design:
The import is not all-or-nothing (refuted K2). A request that fails on a later entry returns 400 after applying the earlier ones. The applier therefore treats any non-2xx response as a failed run: it keeps the previous hash, exits non-zero and retries on the next schedule. Because every import is an upsert, the next good import converges.
Every restart re-applies the boot file over anything applied live. The applier and the boot import must keep reading the same rabbitmq-definitions Secret, so a restart can never roll back a live change.

The alternative is to keep boot-only definitions and batch every principal change into one approved restart window per wave. That means about four production broker restarts (staging, production, regex tightening, retirement).

5. Secret distribution¶

Passwords¶

The existing pattern (#1430) stays: a CreatedOnce ExternalSecret per principal in rabbitmq, generated by the rabbitmq-principal-password ClusterGenerator (48 alphanumeric characters). No human sees a workload password, and nothing is stored in git or Secret Manager. Decision D3. The break-glass password is the exception: GCP Secret Manager secret rabbitmq-breakglass (project camarades-net), synced only into rabbitmq by an ExternalSecret through gcpsm-secret-store-rabbitmq. Its custody does not depend on the cluster.

In-cluster consumers¶

One ClusterSecretStore per environment (rabbitmq-syrf-production, rabbitmq-syrf-staging) with conditions.namespaces set to that environment's namespace, and a reader ServiceAccount whose Role may get only that environment's principal Secrets (resourceNames). In each environment namespace, one ExternalSecret per workload produces rabbitmq-api, rabbitmq-pm and rabbitmq-quartz with the key rabbitmq-password, so the syrf-common env.rabbitmq block works unchanged through rabbitMq.authSecretName. The workload switch is then two values (rabbitMq.username, rabbitMq.authSecretName) in syrf/environments/<env>/<service>/values.yaml. Decision D10 covers the store granularity (the contract says one store per principal).

Previews¶

Each preview namespace gets a CreatedOnce ExternalSecret rabbitmq-preview-principal (key rabbitmq-password) from the same generator, rendered by src/charts/preview-infrastructure. The preview reconciler (Phase 2) is a CronJob in rabbitmq that runs as syrf-broker-provisioner. Its ClusterRole may list namespaces and get Secrets named exactly rabbitmq-preview-principal (a ClusterRole's resourceNames applies in every namespace). Every 5 minutes it (one run at a time, concurrencyPolicy: Forbid):

lists namespaces labelled syrf.org.uk/environment: preview (Kubernetes API, not the broker);
makes one GET /api/vhosts?columns=name&disable_stats=true and one GET /api/users?columns=name&disable_stats=true;
skips, for this run only, any preview whose rabbitmq-preview-principal Secret does not exist yet (get returns 404 because ESO has not rendered it). A skipped preview never fails the run;
for each preview whose vhost is missing: PUT /api/vhosts/pr-N, then DELETE /api/permissions/pr-N/syrf-broker-provisioner. RabbitMQ gives the user who creates a vhost over HTTP .* on it (proved on a local 3.13.6 in cluster-gitops#1669), and the provisioner keeps no vhost grants;
for every live preview, on every run: PUT /api/users/syrf-pr-N (password from the namespace's Secret, tags ""), PUT /api/permissions/pr-N/syrf-pr-N (.*) and the ADR-015 vhost limits (PUT /api/vhost-limits/pr-N/max-connections and …/max-queues). All are idempotent upserts, so a changed limit also reaches existing previews. The cost is four small PUTs per preview per run (about twenty today). The broker therefore always holds the Secret's current password. If rabbitmq-preview-principal is ever recreated with a new value, the broker side recovers within one interval, but running pods keep the old value, because they read rabbitmq-password from the environment at start-up. Their open connections survive until the next reconnect, which then fails with ACCESS_REFUSED. Recovery therefore also rolls the preview's API, PM and Quartz Deployments (Argo's Restart action on the preview app, or a redeploy of the preview). A Helm checksum annotation cannot do this automatically: Helm never sees an ESO-generated value;
for each pr-* vhost or syrf-pr-* user whose namespace is gone: DELETE the user, then the vhost.

The admin credential never enters a preview namespace. The rabbitmqctl step in _preview-cleanup.yml is retired (syrf#4251). Preview workloads that start before their vhost or user exists retry until it does: MassTransit reconnects, and the Preview-Ready contract (ADR-016) stays fail-closed.

How a preview's workloads switch (Phase 2 step 3, syrf#4310). The preview ApplicationSet renders every chart, preview-infrastructure included, at the PR's head SHA, so the switch lives in the charts, not in ApplicationSet values. In a preview (environment.name: preview) with rabbitMq.previewPrincipal.enabled true (the chart default), syrf-common's env.rabbitmq block gives the API, PM and Quartz username syrf-<namespace> (syrf-pr-N, the reconciler's syrf-$ns) and Secret rabbitmq-preview-principal key rabbitmq-password, and the API renders no PreSync vhost Job. Any head that contains this switch also contains the Secret (syrf#4236) and the API hook switch (syrf#4199), so a preview moves only when its branch includes syrf#4310 and redeploys; until then the reconciler skips it (no Secret) and it keeps rabbit and the PreSync hook. Precedence while the switch is on: rabbitMq.username, the username key of rabbitMq.configMapName and rabbitMq.authSecretName are ignored for the broker credential (chart defaults cannot be told apart from explicit values), and rabbitMq.provisionPreviewVhost is ignored; an env: map entry still wins over generated env vars. To use any other credential in a preview, set rabbitMq.previewPrincipal.enabled: false (the cluster-gitops rollback lever). Staging, production and local renders are byte-identical before and after. Helpers: src/charts/syrf-common/templates/_rabbitmq-preview-principal.tpl.

s3-notifier Lambdas¶

MVP (no new infrastructure): keep the chart's env-vars-job, pointed at the per-environment consumer Secret (rabbitmq-s3notifier, as #1433/#1434 already do; previews use rabbitmq-preview-principal). Two gaps stay open in the MVP and are documented:

The PostSync dependency. The job runs only when the Argo app syncs, so a changed Secret never reaches the Lambda by itself. Every switch and rotation change must re-sync that environment's notifier env-vars in the same change (hard rule; the 2026-10-14 incident in §1 lost production notifications for about five and a half hours when it did not). Every such PR must say "then sync s3-notifier-<env>", and the old principal must stay valid until the sync is verified. The rotation model (new principal, switch, delete old) makes that safe.
Plain text in the Function CR. ACK mirrors the Lambda configuration into spec.environment.variables of every Function CR (R5), and Argo shows live manifests. Anyone with get on those CRs, or Argo app access to the s3-notifier-* apps, can read a notifier password.

Target (decision D9): the Lambda reads its password from AWS Systems Manager Parameter Store (SecureString, /syrf/<env>/rabbitmq/s3notifier) at cold start, caches it, and re-reads it once on an authentication failure. The env var holds only the parameter name, and the Lambda role gets ssm:GetParameter on that one ARN (plus kms:Decrypt on the AWS-managed key). The parameter is written from the in-cluster principal Secret by an ESO PushSecret through an AWS provider authenticated by the existing pod-identity-webhook. Rotation then needs no Argo sync. This needs a notifier code change, an IAM change and a new ESO store, so it is a follow-up and not on the critical path.

ARRNC PDF agents¶

Unchanged from the contract: the -pdfagent value is copied out of band into the slot's server-config age scope (staging) and recorded without the value. How the production slot will receive its credential is U (no reference in arrnc-api-deploy on origin/main); it must not be rabbit.

6. Management UI exposure¶

Users of 15672 today¶

User	Path	Credential	Fate
API preview PreSync hook	in-cluster `rabbitmq.rabbitmq.svc:15672`	`rabbit`	not rendered in a preview whose head includes syrf#4310 (Phase 2 step 3); gone once no older head is deployed
Local `rabbitmq` MCP server (`amq-mcp-server-rabbitmq`, `~/.config/mcp-proxy/config.json`, not in git)	public `rabbitmq.camarades.net` (no longer routed since #1639)	`rabbit`, `--allow-mutative-tools`	removed (decision D5)
People	public Ingress on 443 (removed by #1639)	`rabbit`	port-forward with the observer (cluster-level metrics and per-vhost connections only; no user or permission reads), or the provisioner/break-glass with Chris's go
`docs/platform/masstransit-observability.md`	documents `kubectl port-forward svc/rabbitmq 15672:15672 -n syrf` (the namespace is wrong; it is `rabbitmq`)	none	corrected (namespace `rabbitmq`, observer or approved break-glass)

Nothing else in syrf, cluster-gitops or camarades-infrastructure references 15672 or /api/ on the broker (V, git grep). Metrics use 9419 (Prometheus plugin, no credential). The ingress's GCE annotations (ingress.kubernetes.io/forwarding-rule and others) are stale: the only forwarding rule in camarades-net is the ingress-nginx one (34.13.63.21, ports 80–5672), so there is no second public path (V).

Target (decision D4)¶

No Ingress for the management UI. Operators use kubectl -n rabbitmq port-forward svc/rabbitmq 15672 with the observer, or with break-glass when Chris approves. Any broker API call still needs Chris's go (Hard operational rule). Holding cluster credentials is the access boundary.
Give DNS an owner. Done 2026-10-15 (cluster-gitops#1662). The annotation external-dns.alpha.kubernetes.io/hostname: rabbitmq.camarades.net is on the ingress-nginx controller Service, so External-DNS owns the name that AMQPS clients use and follows an ingress-nginx IP change. Verified live: the annotation is present, the IP is still 34.13.63.21, and openssl s_client -connect rabbitmq.camarades.net:5671 gives verify 0. (#1639 had landed the Ingress removal first, so for about a day the record held only because External-DNS is upsert-only.)
Certificate renewal still works. rabbitmq-amqps-tls uses HTTP-01 through the nginx class. cert-manager creates its own temporary solver Ingress, so it does not need the management Ingress. The phase still verifies one renewal (cmctl renew in a window, or the next natural renewal) before it is closed. The old Ingress Secret rabbitmq-tls is deleted afterwards.
Alternatives: an IP allow-list annotation (nginx.ingress.kubernetes.io/whitelist-source-range), which is fragile with home IPs and still exposes the login page to the allowed ranges; or oauth2-proxy in front, the pattern used for docs-production, which is more moving parts for a UI that only operators use.

7. Phases¶

Each phase is its own PR or PR set. Dependencies:

Phases 0 and 6 depend on nothing and can start at once.
Phase 1 needs the Phase 0.4 proofs.
Phases 2 (previews) and 3 (staging) need Phase 1 and can run in parallel. Phase 2 also needs Phase 0.6. Each environment's regex-tightening step needs the Phase 0.1–0.3 fixtures.
Phase 4 (production) needs Phase 3 verified. Phase 5 needs Phases 2–4 and its 7-day entry proof.

The environment order follows the brief: previews, then staging, then production.

Phase 0: code, fixtures and local proofs (syrf; nothing live)¶

#	Work	Acceptance
0.1	Done (syrf#4201). `broker-permissions.json` for `api`, `project-management`, `quartz` and `s3-notifier`. Implemented in syrf#4201; corrections to the drafts in Phase 0.1–0.2 results	AC8
0.2	Done (syrf#4201). Generalise `PdfAgentBrokerPermissionFixture` into `SyRF.Testing.Containers` and add one broker-permission test per service: the production registration under the committed regexes, all paths in §3, plus the negative tests. Implemented in syrf#4201: `RabbitMqBrokerPermissionFixture` in `SyRF.Testing.Common` (namespace `SyRF.Testing.Containers`), used by the PDF agent's tests and by `ApiBrokerPermissionTests`, `ProjectManagementBrokerPermissionTests`, `QuartzBrokerPermissionTests` and `S3NotifierBrokerPermissionTests`. They are `Category=Integration`, so AC8 is met when the main-branch integration lane runs them green	AC8
0.3	Implemented (syrf#4225). E2E stack: every service connects as its own restricted user (extend `bulk-pdf-lane.sh`'s `syrf-e2e-pdfagent` pattern to API, PM, Quartz and the LocalStack notifier). `e2e/scripts/setup.sh` creates `syrf-e2e-{api,pm,quartz,s3notifier,pdfagent}` from each committed `broker-permissions.json` through `scripts/broker-principals.py`, the one renderer later phases reuse for the cluster-gitops definitions (it fails on an unknown or missing placeholder). The smoke and Bulk PDF lanes pass with every service on its own user, and a removed grant is refused visibly (broker log and service log). Finding for Phase 3: MassTransit replaces a `dotnet` process name with the entry assembly's file name (`BusHostInfo`, 8.4.0 and 8.5.11), and the images run `dotnet <dll>`, so in-cluster bus endpoints are `<host>_SyRFAPIEndpoint_bus_<id>` and so on (project management truncates to `syrfprojectman_SyRFProjectMan_bus_<id>`), not `…_dotnet_bus_…`. The in-cluster values in §3 and in the files' `placeholders` descriptions were corrected in syrf#4228, server-config#306 and cluster-gitops#1659	AC9
0.4	Done (syrf#4200). Local RabbitMQ 3.13.6 proofs for §4: live `POST /api/definitions` upsert and tag demotion, no deletion of unlisted users, `DELETE /api/users/rabbit`, and the Bitnami 14.6.6 behaviour when `auth.username` names a user that the definitions no longer list. Probes are already credential-free (V): with `loadDefinition.enabled` the live StatefulSet's readiness is `rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms` and liveness `rabbitmq-diagnostics -q ping`, so demoting or deleting `auth.username` cannot make `rabbitmq-0` unready. The proof re-checks the rendered probes, because disabling `loadDefinition` would switch the chart to `curl --user <auth.username>` probes. Done 2026-10-14 (syrf#4200): local proofs. Probes and start-up confirmed (the chart never re-creates or re-tags `auth.username` while `loadDefinition` is on); refuted: import atomicity (K2) and "removing the entries revokes `rabbit`'s grants" (Phase 5.2); decided: the Phase 5.4 chart change restarts the broker.	gate for Phases 1 and 5
0.5	Done (cluster-gitops#1656). A check that the regexes in the cluster-gitops definitions equal the syrf files at the deployed commit (a script in cluster-gitops CI that fetches the files at the image's `sha`). It reads syrf with the dedicated read-only GitHub App `camarades-syrf-reader` (Contents: read, installed on syrf only; secrets `SYRF_READER_APP_ID` and `SYRF_READER_PRIVATE_KEY`). A `known_gap` ratchet downgrades only a regex mismatch or a missing contract file, and nothing else	AC7
0.6	Done (syrf#4199). API chart: a values switch `rabbitMq.provisionPreviewVhost` (default `true`, so behaviour is unchanged) that the preview ApplicationSet turns off in Phase 2. Implemented in #4199.	AC10

Phase 0 is complete (2026-10-15). Flag decision: not flagged. Phase 0 changes no runtime behaviour, and every later switch is a per-environment values change whose rollback is a revert.

Phase 1: provisioning foundation (cluster-gitops `rabbitmq` namespace) (P)¶

Status (2026-10-17): done. Steps 1 to 4 are live; the provisioner has not yet made an import.

Done. ExternalSecrets for every new principal (§2) with refreshPolicy: CreatedOnce, sync-wave -10 (cluster-gitops#1664, merged 2026-10-15T01:45Z): eight, all SecretSynced.
Done. Added to the definitions template with interim grants: .* on their own vhost and tag "", as #1430 did for the notifiers. syrf-broker-provisioner is administrator with no permissions. syrf-production-pdfagent is on the Paused set (#1663, D11).
The definitions applier (§4), first authenticating as rabbit. First run done 2026-10-15T01:46:27Z: HTTP 204, sha256 8cd85bd7…. Switched to the provisioner by cluster-gitops#1668 (merged 2026-10-15T03:21Z); no import since.
Done. The per-environment ClusterSecretStores and consumer ExternalSecrets (§5): #1688 (staging, including rabbitmq-pdfagent-gke) and #1690 (production), verified 2026-10-17 without a broker query.
Verify (no broker query): the 8 ExternalSecrets are SecretSynced; the rendered load_definition.json parses (checked without printing values); the applier's logged result is HTTP 204 with the expected sha256 (a 4xx would mean the import failed part-way); the applier Job and CronJob succeed; rabbitmq-0 start time and restart count are unchanged; no workload logs a broker auth or connection error. No workload uses the new principals yet. A per-user permission read is not part of the default evidence: it needs the provisioner and Chris's go for that call (the observer gets 401 on it). Steps 1 to 3 (first run) were verified this way on 2026-10-15 (§1).
Rollback: revert. Users created live stay until Phase 5 deletes them, but they are inert: nothing holds their passwords outside rabbitmq.

Phase 2: previews (after Phase 0.6) (P) for steps 1, 2 and 4¶

The preview reconciler (§5) owns preview vhosts and principals, so this phase deploys it first, then switches. The reconciler runs as an administrator and writes to the broker that production shares, so deploying it, turning off its dry run and changing the shared ClusterExternalSecret each need Chris's go.

Deploy the preview reconciler (§5) with dryRun: true: it logs the PUTs and DELETEs it would make. Done: cluster-gitops#1669 (CronJob rabbitmq-preview-reconciler, DRY_RUN: "true", limits max-connections 50 and max-queues 256) and syrf#4236 (the per-preview rabbitmq-preview-principal ExternalSecret, inert until step 3). The dry run records absences in its state ConfigMap, so its would DELETE lines are what the first live run deletes, including vhosts left by previews that closed before the reconciler existed.
Turn off dry run. Each preview gets syrf-pr-N. Remove the rabbitmqctl vhost deletion from _preview-cleanup.yml; the reconciler deletes a closed preview's user and vhost once its namespace is gone. Done (Chris's go, 2026-10-16): cluster-gitops#1681 (DRY_RUN: "false") merged at 08:01Z; the first live run (08:02Z) deleted the 8 vhosts below, each HTTP 204, with 0 errors, and rabbitmq-0 kept 0 restarts. syrf#4251 then removed the _preview-cleanup.yml step. Its last dry run would delete 8 vhosts whose namespaces are gone though their PRs are still open (pr-2224, 2322, 2461, 2469, 2528, 2784, 2786, 2802); Chris approved that, since the API PreSync hook recreates a vhost on redeploy. Until a preview redeploys from a head that includes syrf#4236 (which renders its Secret), the live reconciler provisions nothing for it. preview-sweep.yml "Layer 4a", the last automated rabbitmqctl use (AC12), was retired by syrf#4274 (Chris's go, 2026-10-17).
Switch the preview workloads, per head. API, PM and Quartz: syrf#4310 (chart switch rabbitMq.previewPrincipal.enabled, default true; see §5 Previews). A global ApplicationSet values change is not used: the ApplicationSet renders the charts at each PR's head SHA, and a head older than syrf#4236 has no rabbitmq-preview-principal Secret (its pods would fail with CreateContainerConfigError) and, older than syrf#4199, an API PreSync hook that would log in as syrf-pr-N with rabbit's password and fail the sync. A preview therefore moves when its branch includes syrf#4310 and redeploys; previews on older heads stay on rabbit (on 2026-10-17 all five live previews, pr-2412, pr-3292, pr-3327, pr-3394 and pr-3934, had heads from 2026-09-06 to 2026-10-02). No cluster-gitops change is needed to switch; cluster-gitops adds the ApplicationSet parameter rabbitMq.previewPrincipal.enabled=false only to roll back. Notifier: separate. The preview notifier values point at the same Secret, and #1435 is rebased to use syrf-pr-N instead of rabbit (one manual Argo sync per pr-N-s3-notifier), in its own windowed change; until then the preview notifier keeps rabbit.
Stop copying rabbit-mq into preview namespaces (ClusterExternalSecret selector). Do this last, so a rollback of step 3 still finds the Secret. It waits for both halves of step 3: the notifier on syrf-pr-N (#1435), and no deployed preview left on a head older than syrf#4310 (any such preview still reads rabbit-mq).
Verify step 3 (per preview, no broker query): for the first preview redeployed from a head with syrf#4310, its rabbitmq-preview-principal ExternalSecret is SecretSynced; the next reconciler run logs that preview as provisioned with errors 0; the API, PM and Quartz pods are Ready, read syrf-pr-N, and log no ACCESS_REFUSED; the API app has no rabbitmq-setup PreSync Job; previews on older heads are unchanged; rabbitmq-0 keeps its start time and 0 restarts.
Verify: a new preview reaches Preview-Ready with no rabbit-mq Secret in its namespace; a preview reference upload lands (Lambda CloudWatch log shows the publish, the search import progresses); closing a preview removes its user and vhost (the reconciler's logged teardown result; a direct GET /api/vhosts/pr-N only with Chris's go, as the provisioner).
Rollback: revert step 4 first (the rabbit-mq copies come back within a minute), then step 3: for API, PM and Quartz add the ApplicationSet parameter rabbitMq.previewPrincipal.enabled=false in cluster-gitops (every preview on a head with syrf#4310 re-renders with rabbit and the PreSync hook; older heads ignore the value), and revert #1435 for the notifier (previews go back to rabbit, whose permissions the reconciler does not touch, and the PreSync hook runs again).

Phase 3: staging¶

Order (the contract's B1.8): s3-notifier and reconciler (#1433, already written), staging Kubernetes PDF agent (D6), Quartz, PM, API. One commit per workload; verify each before the next.

Each switch: values rabbitMq.username: syrf-staging-<w> and rabbitMq.authSecretName: rabbitmq-<w> → Deployment rolling update (maxSurge 1, maxUnavailable 0 at one replica). Hard rule: the same change re-syncs the staging notifier env-vars (s3-notifier-staging and its reconciler) wherever the credential the notifier holds changes, and the Lambda configuration is checked before the old credential is removed (K3, D9).
Verify each: Argo Healthy; the new pod Ready; logs show the bus started and no ACCESS_REFUSED; the workload's own start-up log names the new user and the ssl state, which is the evidence for "no rabbit connection from that workload" (a direct GET /api/vhosts/syrf-staging/connections?columns=user,ssl only with Chris's go, as the provisioner, one call); one full cycle of the affected scheduled work (session idle/liveness timers, the daily statistics run for Quartz and PM).
Then tighten: replace the interim .* with the fixture-proven regexes from Phase 0, one principal per commit (applier, no restart). Verify the same way over 24 hours, including a fault path if organic traffic produces one (_error queues get created lazily).
Rollback: revert the values commit (back to rabbit, which keeps its permissions until Phase 5) or revert the regex commit.

Phase 4: production (P) every step¶

Same order and mechanics as staging, each step its own go. Hard rule for every step: if the change alters a credential the production notifier holds, re-sync s3-notifier-production in the same change and check the Lambda's RabbitMqUsername before the old credential is removed (K3, D9; the 2026-10-14 incident).

Notifier: #1434 (AMQPS 5671 and syrf-production-s3notifier), then a manual sync of s3-notifier-production. Verify with the first organic production upload (or a Chris-approved upload into a test project): the Lambda's CloudWatch log shows the publish and no authentication error, and the search import job advances. Confirm one ACK resync later (aws lambda get-function-configuration --query Environment.Variables.RabbitMqUsername) that ACK did not revert the variables (risk K3).
Quartz, 3. PM, 4. API: as staging. Recommended window: 05:00–07:00 UTC. Each rollout has no expected downtime (§8).
Tighten to the proven regexes, one principal per commit, after 7 clean days.
Rollback: revert the values commit; Argo rolls the Deployment back to rabbit. For the notifier, revert and sync again. rabbit stays valid until Phase 5.

Phase 5: retire `rabbit` (P)¶

Entry proof (7 days, including a daily-statistics run): every workload connects as a new principal (each workload's start-up log and the Deployments' and Lambdas' configured usernames; a direct per-vhost connection read only with Chris's go, as the provisioner, one call per vhost); no ACCESS_REFUSED in broker or workload logs; the owners of the local and / vhosts are identified or retired (D12); no Kubernetes object references rabbit-mq outside rabbitmq (kubectl get deploy,sts,job,cronjob -A -o json filtered for secretKeyRef.name == "rabbit-mq").

Stop distributing rabbit-mq to syrf-staging and syrf-production (ClusterExternalSecret selector). No restart: no pod reads it any more.
Demote: in the definitions, rabbit gets tags "" and an explicit ^$ ^$ ^$ entry on each of its four vhosts (/, syrf-production, syrf-staging, local), applied live by the applier. Removing rabbit's entries instead would leave its .* grants in place, because an import never revokes (Phase 0.4, C11). The ^$ entries still let rabbit open those vhosts but not touch any resource; step 5 removes the user. They also keep the rollback a plain revert, which restores .*. The broker's probes do not authenticate (Phase 0.4, confirmed), so this cannot affect rabbitmq-0 readiness while auth.username still names rabbit. From this point the management UI and API refuse rabbit. The applier must already run as the provisioner (Phase 1.3).
Rotate: add a new camarades-rabbit-mq version in GCP SM. ESO re-renders the definitions and the applier sets the new hash. No workload restarts because none uses rabbit. This replaces #1437/#1438 (merged 2026-10-14, §1). Hard rule: re-sync each environment's notifier env-vars in the same change and verify them before the old password is gone; by this phase no notifier should hold rabbit at all, and the entry proof must show it.
Break-glass: create rabbitmq-breakglass in GCP SM, sync it into rabbitmq, add syrf-broker-breakglass (administrator, no permissions) to the definitions, and point the Bitnami auth.username/existingPasswordSecret at it. This restarts the broker (Phase 0.4, C16): the change alters the StatefulSet's pod template (RABBITMQ_USERNAME, the RABBITMQ_PASSWORD secretKeyRef and the checksum/config annotation). It needs a window (a 2 to 5 minute AMQP outage for every environment) or rides with another planned restart. With loadDefinition on, the chart never re-creates or re-tags auth.username at start-up, so only the definitions decide the break-glass user's state. This step must land before step 6 deletes rabbit-mq: the StatefulSet reads RABBITMQ_PASSWORD from it through a non-optional secretKeyRef, so the next broker pod could not start without it.
Delete: remove rabbit from the definitions, then run one approval-gated Job that makes a single DELETE /api/users/rabbit as the provisioner. The order matters: while the file still lists rabbit, the next import or restart re-creates it. The delete force-closes any connection rabbit still holds (CONNECTION_FORCED), which the Phase 5 entry proof must show is none (workload logs and configured usernames; a direct GET /api/vhosts/<vhost>/connections?columns=user per vhost only with Chris's go, as the provisioner, one call at a time, as the pre-delete gate). Remove the local and / grants (D12).
Clean up copies: disable every camarades-rabbit-mq version and delete the secret after 30 days; delete the RABBITMQ_PASSWORD repository secrets in syrf and camarades-infrastructure, and remove the dormant Terraform rabbitmq_* variables and legacy preview-Lambda module; delete the stale rabbitmq-load-definition Secret that #1431 kept as a rollback target; drop rabbit-mq from the definitions template's data and delete the rabbit-mq ClusterExternalSecret.
Rollback: before step 5, revert the definitions commit (the applier restores administrator and the grants). After step 5, recovery uses break-glass to recreate a principal, never a shared admin for workloads (contract B2).

Phase 6: close the management UI (P) (independent; step 1 done)¶

Status (2026-10-15): steps 1, 3 and 4 are done; step 2 and the certificate-renewal proof remain. cluster-gitops#1639 (merged 2026-10-14 04:19Z) set ingress.enabled: false first; the record kept resolving to 34.13.63.21 because External-DNS runs with --policy=upsert-only, but had no owning source. Step 1, cluster-gitops#1662, restored ownership on 2026-10-15.

Done (cluster-gitops#1662). ingress-nginx controller Service ingress-nginx/ingress-nginx-controller carries the External-DNS hostname annotation for rabbitmq.camarades.net. Verified live: the annotation is present, the record is unchanged at 34.13.63.21, and openssl s_client -connect rabbitmq.camarades.net:5671 gives verify 0.
Decide the MCP server's fate (D5) and update ~/.config/mcp-proxy/config.json (local, not in git).
Done via cluster-gitops#1639. Set ingress.enabled: false in the broker values. Bitnami 14.6.6 renders the Ingress separately from the StatefulSet, so the broker does not restart (confirm with a helm template diff in the PR).
Done in the Phase 0.4 PR. Fix docs/platform/masstransit-observability.md (namespace, observer instead of default credentials).
Verify: AC4.
Rollback: step 1 reverts cleanly (the record stays under upsert-only, but loses its owner again). Reverting #1639 brings the Ingress and its certificate back, which reopens the public UI; do that only to restore operator access in an emergency, and prefer port-forward.

8. Production impact per phase¶

Phase	Broker restart	Workload restarts in production	Expected downtime	Needs Chris's go
0	none	none	none	no (syrf PRs; normal review)
1	none with the applier (D1; confirmed, `rabbitmq-0` 0 restarts after the first import); one window (~2–5 min AMQP outage, all environments) without it	none	none	yes: it writes users to the shared broker
2 (previews)	none	none	previews only	yes for the reconciler (an administrator writing to the shared broker) and the ClusterExternalSecret change; no for the preview values
3 (staging)	none	none	staging rollouts only	no for values; yes for regex commits, because the applier writes to the shared broker
4 (production)	none	each switch rolls one Deployment (surge first: the new pod is Ready before the old one stops); the notifier step reconfigures the Lambda	none expected; a few seconds of MassTransit reconnect per service, and in-flight scheduled deliveries retry	yes, every step
5 (retire)	one, at step 4: the Bitnami `auth.username`/`existingPasswordSecret` change restarts the pod (Phase 0.4)	none	~2–5 min AMQP outage at step 4, in a window; none for the other steps	yes, every step
6 (UI)	none (confirm by render diff)	ingress-nginx controller config reload only, no rollout	none for AMQP or web	yes

9. Acceptance criteria¶

#	Criterion	Verification
AC1	No workload holds an administrator credential.	Every API/PM/Quartz/PDF-agent Deployment's `RabbitMqConfig__Username` is a principal from §2 (`kubectl get deploy -A -o json`, value field only); every Lambda's `RabbitMqUsername` likewise (`aws lambda get-function-configuration`); the rendered definitions give those users tags `""`; `kubectl get secret rabbit-mq -A` lists only `rabbitmq` (Phase 5.1) and then nothing (Phase 5.6).
AC2	Each principal is refused outside its vhost and its regexes.	Fixture negative tests in CI (Phase 0.2). Live: the applier's logged `204` for the import of the committed strings, plus the Phase 0.5 parity check, is the default evidence. A per-principal `GET /api/users/<name>/permissions` shows exactly one vhost and the committed strings, but needs the provisioner and Chris's go for that call (the observer gets 401 there). No live negative probes.
AC3	Production notifications still arrive after an upload.	After Phase 4.1: the first organic (or approved test) production upload shows the Lambda publish in CloudWatch with no auth error, and the search import job advances in the UI. Repeated after Phase 4.5 and Phase 5.
AC4	The management UI is not reachable from the Internet.	From an external host: `curl -skS -o /dev/null -w '%{http_code}' https://rabbitmq.camarades.net/` returns the nginx default backend 404, and `curl -skS https://rabbitmq.camarades.net/api/overview` shows no RabbitMQ response; `kubectl get ingress -n rabbitmq` is empty; `openssl s_client -connect rabbitmq.camarades.net:5671 -verify_return_error` still verifies (DNS kept).
AC5	`rabbit` no longer exists anywhere.	Git shows `rabbit` removed from the definitions, the delete Job's logged `204`, and a refused login in the workload logs; a direct `GET /api/users/rabbit` → 404 only with Chris's go, as the provisioner; `git grep -n 'rabbit-mq\\|username: rabbit'` in cluster-gitops `origin/main` finds no workload reference; `camarades-rabbit-mq` versions disabled; `gh secret list` in `syrf` and `camarades-infrastructure` shows no `RABBITMQ_PASSWORD`.
AC6	Exactly two administrators remain, and only in `rabbitmq`/GCP SM.	The rendered definitions (checked without printing values) give `administrator` only to the provisioner and break-glass; a direct `GET /api/users?columns=name,tags&disable_stats=true` (a short list, not a queue listing) only with Chris's go, as the provisioner (the observer gets 401); `kubectl get secret -A` shows their Secrets only in `rabbitmq`.
AC7	Live regexes equal the committed contract.	The Phase 0.5 check passes on cluster-gitops `main`.
AC8	Each service runs under its own regexes.	Each service's broker-permission test passes (positive and negative) in CI on `main`.
AC9	The whole stack works under least privilege.	`run:e2e-smoke` and the Bulk PDF lane pass with every service on its own restricted user.
AC10	Previews are provisioned, repaired and torn down without an admin credential in the preview.	A new preview is Preview-Ready with `syrf-pr-N`; its namespace has no `rabbit-mq` Secret; in one preview, deleting `rabbitmq-preview-principal` (ESO recreates it with a new value), waiting one reconciler interval and rolling the Deployments returns the preview to Ready with no `ACCESS_REFUSED`; after close, the reconciler's logged teardown result shows the user and vhost deleted (a direct `GET /api/vhosts/pr-N` → 404 and `GET /api/users/syrf-pr-N` → 404 only with Chris's go, as the provisioner).
AC11	Phases 1–4 restart no broker (if D1 is accepted).	`rabbitmq-0` pod start time and `restartCount` unchanged across those phases (`kubectl`, no broker query). Phase 1's first import: start 2026-10-14T04:03:34Z, 0 restarts.
AC12	No automation runs `rabbitmqctl` against the shared broker.	`git grep -n rabbitmqctl` over syrf `.github/` and cluster-gitops finds no use against `rabbitmq-0`. Met 2026-10-17 (syrf#4274): only comments remain.
AC13	Rotation needs no coordinated restart.	One principal is rotated by the new-principal/switch/delete method in staging, with no broker restart and only the switched workload rolling.
AC14 (D9 only)	No notifier password in a Function CR.	`kubectl get functions.lambda.services.k8s.aws -A -o json \\| jq '.items[].spec.environment.variables \\| keys'` shows no `RabbitMqPassword`.

10. Wolverine and the MassTransit 8.5.11 bridge¶

8.5.11 (syrf#4147). Merged before Phase 0.2; the broker-permission tests (syrf#4201) run on 8.5.11. Any later MassTransit bump must run them green.
Wolverine (ADR-021, single cut-over). The principal model, vhosts, secret plumbing, applier, reconciler and break-glass do not change. The regexes do: ADR-021 S1.4 and S2.9 already require them to be regenerated from the Wolverine topology and proved by the same fixture. What changes:
MassTransit names (Namespace:Type exchanges, quartz, Job*, MassTransit.*, _error/_skipped, <host>_<process>_bus_<id>) disappear. Wolverine names its own queues and exchanges, its reply queues and, with durability, its control queues; automatically named reply queues need configure. Pin their names so the regexes stay exact.
Scheduling moves from the Quartz service to Wolverine's database-backed scheduling, so -quartz and its confused-deputy allowlist go away. The principal is deleted after the cut-over.
Opportunity: Wolverine can run with automatic provisioning off. If PM provisions the topology and the clients (Lambda, PDF agent, API) only send, the clients need write only. That closes the configure-implies-delete residual (D6) and the read-implies-consume residual (D14) for free.
Because the switch-over is a single cut-over, the regex change ships in the same window: a definitions commit applied live (D1) just before the switch, with the MassTransit regexes kept until rollback is no longer possible.

11. Decisions for Chris¶

Accepted 2026-10-14: all fourteen as recommended (Chris: "all as recommended"). D4's Ingress removal landed through cluster-gitops#1639 and the DNS owner through cluster-gitops#1662 (2026-10-15); see Phase 6. D11 is done (#1663). D13 is moot (see §1).

ID	Decision	Recommendation
D1	How principal changes reach the broker	A live definitions applier (one `POST /api/definitions` per change) using a standing `syrf-broker-provisioner` admin in `rabbitmq` only. It removes about four production broker restarts. Alternative: boot-only definitions with batched restart windows. The Topology Operator (the contract's original D1) would need its admin connection Secret next to every CR, so it either spreads the admin credential to preview namespaces or needs the same cross-namespace plumbing.
D2	Preview principals	*Per preview (`syrf-pr-N`, `.` on `pr-N`)**, owned by the reconciler, as the contract specifies. Fallback: one shared `syrf-preview` principal granted on each `pr-N` (simpler, but previews can read each other).
D3	Where principal passwords come from	Keep the in-cluster generator (status quo since #1430). GCP SM only for break-glass.
D4	Management UI	Remove the Ingress; port-forward only, with DNS ownership on the ingress-nginx Service. The Ingress removal landed through #1639 before the DNS step; the record has an owner again since #1662 (2026-10-15). Remaining: the certificate-renewal proof.
D5	The local `rabbitmq` MCP server	Remove it from the shared gateway. Its queue listing tools can make exactly the unfiltered calls that caused the 2026-10-10 OOM, and it runs with mutative tools as `rabbit`. If it is wanted later: observer credential, port-forward, no mutative tools, vhost-scoped calls only.
D6	Staging Kubernetes PDF agent (`Enabled`, as `rabbit`; ADR-015 says no agent runs in Kubernetes)	Give it its own principal `syrf-staging-pdfagent-gke` (Enabled set) now so behaviour is unchanged, and retire it under ADR-015 separately. It must not share `syrf-staging-pdfagent`, which holds the ARRNC slot's Paused set.
D7	One step or two per environment	Two: first the identity switch with `.*` on the principal's own vhost (removes admin and cross-environment reach), then the proven regexes. Each is a separate revert.
D8	In-cluster AMQPS	Not in this plan. Keep in-cluster clients on 5672 while principals move; switching the transport at the same time doubles failure modes and changes persisted Quartz destination addresses. In-cluster AMQPS (contract B1.0 `TlsServerName`) and closing 5672 (contract B2.3) stay separate.
D9	Notifier credential delivery	MVP: keep the env-vars job with a documented manual sync per switch. Follow-up: SSM Parameter Store `SecureString` read at cold start, written by ESO `PushSecret` (§5).
D10	Secret store granularity	One ClusterSecretStore per environment, restricted to that namespace and to that environment's principal Secret names. This deviates from the contract's one store per principal: the namespace is already the boundary, because any pod that can read Secrets in it can read every Secret there.
D11	`syrf-production-pdfagent`	Set it to the Paused set now (it has `.` on production and nothing should use it until the ARRNC production cutover). Done:* cluster-gitops#1663, live with the 2026-10-15T01:46Z import; no `syrf-pdf-agent-production` container exists on arrnc-api (confirmed 2026-10-15).
D12	The `local` and `/` vhosts (contract D4)	Retire them in Phase 5 unless an owner is found by then. Developer uploads to the production bucket with `virtualhost=local` already fail after #1434, as intended.
D13	#1437 and #1438	Close both as superseded by this plan (§12). Moot: another session merged both on 2026-10-14 around 04:00Z; see the incident.
D14	`queue:` senders that can consume the queues they send to (`-api`, `-s3notifier`)	Accept for the MVP: the API and the notifier are first-party, and the main gain is no admin and no cross-environment reach. Follow-up: PM-owned ingress exchanges and `exchange:` sends, as #4138 does for the PDF agent.

cluster-gitops#1437 and #1438 were meant to be closed as superseded by this plan, but another session merged them around 04:00Z on 2026-10-14 (D13 is moot; the predicted failure happened, see the incident). They rotated the shared rabbit password while every workload still uses it. #1438 restarts the broker and the in-cluster consumers, then re-syncs only the preview notifier jobs. The production and staging notifier Lambdas kept the old password, so production upload notifications stopped until s3-notifier-production was re-synced at 09:43Z. Under this plan, rabbit is rotated only after nothing uses it (Phase 5.3), with no workload restarts.
cluster-gitops#1434 (production notifier on syrf-production-s3notifier over AMQPS) is Phase 4.1 as written.
#1433 is the first step of Phase 3. #1435 is rebased in Phase 2 to use syrf-pr-N.
#1436 (close public 5672) belongs to the transport track. Its gate (every Lambda and ARRNC agent on 5671) is met after Phases 2–4.
syrf#4139 and #4140 (PDF agent ingress exchange) are compatible: the PM draft already allows .Agent.

13. Risks¶

#	Risk	Mitigation
K1	A regex gap appears only on a rare path (a fault creates `_error` lazily; a scheduled redelivery fires days later).	Phase 0.2 forces fault and redelivery paths; D7's `.` interim step separates identity from narrowing. Consumers* use `UseInMemoryOutbox`, so a refused publish inside a consumer causes redelivery, not loss. Publishers outside a consumer have no outbox: API controllers and handlers, `NotificationHub`'s `SchedulePublish`/`CancelScheduledPublish`, PM's recurring `*Schedule` services, and the notifier Lambda's `Publish`, `Send` and `SubmitJob`. There a refused operation fails the HTTP request, the hub call or the schedule registration, and for the Lambda it fails the invocation: S3 retries an asynchronous invocation twice, after which a reference-upload notification is lost (bulk-PDF objects are also covered by the reconciler's durable capture in staging). The Phase 0.2 fixtures must therefore drive each of these publisher paths, not only consumers.
K2	The applier imports a broken file (for example after a template error).	ESO keeps the last good render on failure; the applier skips unchanged hashes and fails closed on JSON parse errors. The import is not all-or-nothing (Phase 0.4 refuted it, C13): a request that fails on a later entry returns 400 after applying the earlier ones, while malformed JSON is rejected whole. The applier therefore records the hash only after a 2xx response and retries otherwise; every import is an upsert, so the next good import converges.
K3	ACK reverts the Lambda environment set by the env-vars job, or a credential changes while a notifier still holds the old one. The second form happened on 2026-10-14: the production notifier received the rotated `rabbit` password only at 09:43Z, and every production upload notification from about 04:00Z failed with "Broker unreachable" (one search import lost).	Evidence says ACK does not revert: post-creation bulk-PDF keys appear in the staging CRs, so the CRs follow AWS. Phase 3 and 4.1 still confirm the username one resync later. Hard rule (D9): every later credential switch re-syncs each environment's notifier env-vars in the same change and verifies the Lambda configuration before the old credential is removed (Phases 3 to 5).
K4	The reconciler deletes a live preview's vhost (for example if the Kubernetes API list fails), or a recreated preview Secret locks the preview out.	It deletes only when the namespace list call succeeded and the namespace is absent from two consecutive runs; dry-run first; `concurrencyPolicy: Forbid`. The user's password and permission are re-applied on every run (§5), so the broker side heals within one interval; the preview's Deployments are then rolled (AC10).
K5	The provisioner credential, or the reconciler's Kubernetes identity, leaks.	The provisioner password exists only in `rabbitmq` (generator, CreatedOnce). It is read by the applier's and reconciler's ServiceAccounts, and it is also rendered into the ESO-built `rabbitmq-definitions` Secret that the broker pod `rabbitmq-0` loads. The reconciler's ClusterRole is broader than one Secret: it may `list` namespaces and `get` `rabbitmq-preview-principal` in every namespace, so its identity can read every preview's password (never a staging or production one, which have different names). That ServiceAccount is used only by the reconciler CronJob in `rabbitmq`. Rotation by the new-principal method; the provisioner is listed in AC6.
K6	Certificate renewal is unproven without the management Ingress. (The DNS-ownership half is closed: cluster-gitops#1662 gave `rabbitmq.camarades.net` an owner on 2026-10-15; verified live, IP `34.13.63.21`, verify 0 on 5671.)	Renewal is proved (`cmctl renew` in a window, or the next natural renewal) before Phase 6 closes. cert-manager's temporary solver Ingress does not need the management Ingress.
K7	Somebody queries the shared broker during a rollout, or verifies with an unfiltered listing.	Chris's standing rule: no direct broker queries; verify from the applier's logged result, ExternalSecret and Job status, and workload logs. The hard rule is repeated in every phase PR's verification section; D5 removes the easiest way to do it.

14. Follow-ups outside this plan¶

In-cluster AMQPS and closing 5672 (contract B1.0/B2.3; D8).
Notifier secret delivery via SSM (D9).
Ingress exchanges for API and notifier sends (D14).
Retire the Kubernetes staging PDF agent under ADR-015 (D6).
Deliver the production ARRNC slot's credential through a recorded channel, not a workflow -e flag.

Replace the shared rabbit administrator with least-privilege broker principals¶