Broker definitions: local RabbitMQ 3.13.6 proofs (plan Phase 0.4)¶

This is the evidence for Phase 0.4 of the broker principals plan. It tests the claims in the plan's §4 and the Bitnami chart questions that Phases 1 and 5 depend on. Every command and its output is below.

Nothing here touched the shared broker. No kubectl, no port-forward, no MCP broker tool and no request to rabbitmq.camarades.net or the cluster Service. Every broker call went to 127.0.0.1 on a throwaway local Docker container named syrf-p04-*. All of them, and their volumes, were removed afterwards (docker ps -a --filter name=syrf-p04 and docker volume ls both returned nothing). Reading the cluster-gitops values through the GitHub API and rendering the chart with helm template were the only remote reads.

Contents¶

Results
What changes in the plan
Set-up
A. Live definitions import on rabbitmq:3.13.6-management
H. Bitnami chart 14.6.6 render with the real values
B. Bitnami start-up behaviour in Docker
Appendix: AMQP probe

Results¶

Each claim is quoted or paraphrased from the plan. "Confirmed" and "refuted" refer only to RabbitMQ 3.13.6 and Bitnami chart 14.6.6 as tested here.

#	Claim (plan section)	Result	Evidence
C1	The import creates or updates the listed users and permissions and never deletes anything (§4, "Today")	Confirmed, with a gap: it also never revokes a permission whose entry is removed from the file	A2, A3
C2	RabbitMQ applies definitions posted over HTTP immediately, upserting like the boot import; no restart is needed (§4, "Proposed")	Confirmed. `POST /api/definitions` returns 204; the beam PID (`"20"`) and the container start time do not change from A0 to A4	A1, A4
C3	The applier can start as `rabbit`, create `syrf-broker-provisioner` with its first import, then run as the provisioner (§4, "Identity")	Confirmed. A1 posts as `rabbit` and creates the provisioner; A2 to A5 run as the provisioner, including the demotion and deletion of `rabbit`	A1 to A5
C4	The boot import stays; if the PVC is lost, the boot import rebuilds the same state (§4, "Boot import stays")	Confirmed for the state the file describes. Two consequences: every restart re-applies the boot file over live changes (a narrowed grant went back to the file's `.*`; a deleted `rabbit` came back), and objects created only live (a preview user) survive a restart but not PVC loss	A6, B1
C5	A live import with plaintext `password` fields creates users (§4, "Local proof")	Confirmed	A1
C6	It updates the passwords of existing users	Confirmed. The observer's old password gets 401, the new one 200	A1
C7	It updates the tags of existing users	Confirmed (`rabbit`: `administrator` to none)	A3
C8	It changes permissions without a restart	Confirmed. The narrowed notifier and the Paused production agent are refused `queue.declare` (403) straight after the import	A1
C9	Removing an entry deletes nothing	Confirmed for users and for permissions	A2, A3
C10	Changing `rabbit`'s tags to `""` demotes it live	Confirmed for HTTP: the management API answers 401 "Not management user". Tags do not govern AMQP: the demoted `rabbit` still opened `syrf-production` and declared a queue	A3
C11	Phase 5.2: with tags `""` in the definitions, `rabbit` "loses every permission (applier, live)"	Refuted as written. Dropping `rabbit`'s permission entries from the file leaves all four `.*` grants in place. Explicit `^$` entries remove resource access but still let it open the vhost; `DELETE /api/permissions/{vhost}/rabbit` removes vhost access	A3, A3b, A3c
C12	Phase 5.5: one `DELETE /api/users/rabbit` removes `rabbit`	Confirmed. 204, then 404; HTTP 401 and AMQP refused. New: the broker force-closes `rabbit`'s open connections (`320 CONNECTION_FORCED - user 'rabbit' is deleted`). A file that still lists `rabbit` re-creates it on the next import or restart, so the file change must come first, as the plan orders it	A4, A6
C13	Risk K2: the definitions import is all-or-nothing (U)	Refuted. An import that fails on a later entry returns 400 after creating the earlier user. Malformed JSON is rejected before anything is applied	A5
C14	Phase 0.4 row: with `loadDefinition.enabled` the probes are credential-free, so demoting or deleting `auth.username` cannot make `rabbitmq-0` unready; disabling `loadDefinition` switches to `curl --user` probes	Confirmed by the render and by running the rendered probe commands with `rabbit` absent, demoted and deleted	H1, H2, B1, B2
C15	Does the chart's start-up re-create or re-tag `auth.username` when the definitions no longer list it? (Phase 0.4 question)	No, with `loadDefinition.enabled: true` and a file that has a `"users"` key (ours always does). On an empty PVC RabbitMQ logs "Will not seed default virtual host and user: have definitions to load"; on persisted data the Bitnami script only logs "Persisted data detected". It would re-create the user only if `loadDefinition` were off (empty PVC) or the file had no `"users"` key	B1, B2, B3, image source
C16	Phase 5.4: pointing `auth.username`/`existingPasswordSecret` at break-glass "needs a window if the chart change restarts the broker"	Decided: it restarts the broker. The change alters the StatefulSet pod template (`RABBITMQ_USERNAME`, the `RABBITMQ_PASSWORD` secretKeyRef and the `checksum/config` annotation), so the StatefulSet rolls `rabbitmq-0`	H3

One more fact from the render matters for Phase 5.6: the StatefulSet reads RABBITMQ_PASSWORD from the rabbit-mq Secret through a non-optional secretKeyRef (H1). If rabbit-mq is deleted from the rabbitmq namespace while auth.existingPasswordSecret still names it, the next broker pod start would fail with CreateContainerConfigError. That is standard Kubernetes behaviour; it was not reproduced here because no local Kubernetes cluster (kind) was available.

What changes in the plan¶

These edits are made in broker-principals-plan.md in the same PR:

§4 and K2. The applier must treat any non-2xx response as a failed run: keep the previous hash, exit non-zero and retry on the next schedule. A partly applied import is upsert-only, so the next good import converges. The boot import and the applier must keep reading the same rabbitmq-definitions Secret, because a restart re-applies the boot file over anything applied live (C4).
Phase 5.2. Demotion writes rabbit with tags "" and explicit ^$ entries for each of its four vhosts. Removing the entries would leave .* in place (C11). The explicit entries also keep the rollback a plain revert, which restores .*.
Phase 5.4 and §8. Switching auth.username/existingPasswordSecret restarts the broker (C16), so the step needs a window (about 2 minutes of AMQP outage at 18 vhosts, per cluster-gitops#1634) or must ride with another planned restart. It must land before Phase 5.6 deletes rabbit-mq from rabbitmq.
Phase 0.4 row. Marked done, with a link to this page.

Set-up¶

Item	Value
Upstream image	`rabbitmq:3.13.6-management` (`rabbitmq@sha256:b61a8a862766e4c5b526dd0cb0c382b3f006f25b21d9bf7e425ae6c7011ee87b`), RabbitMQ 3.13.6 on Erlang 26.2.5.2
Bitnami image	`docker.io/bitnamilegacy/rabbitmq:3.13.6-debian-12-r1` (`sha256:c6e65c758507a458c7f99e06cde560e0da21c848cdd0473243d8ef9504697246`), the image that chart 14.6.6 renders with the live values
Chart	`bitnami/rabbitmq` 14.6.6, rendered with Helm v3.16.4
Values	`plugins/helm/rabbitmq/values.yaml` on cluster-gitops `main`, last changed by `bdb71e62` (#1639), read through the GitHub API
Definitions	Test files shaped like the ESO template `resources/rabbitmq-definitions.yaml`: the same users, tags, vhosts and grants, with throwaway passwords such as `test-rabbit-1`. No real credential was used.
AMQP checks	A 130-line Python standard-library AMQP 0-9-1 client (appendix) that logs in, opens a vhost and optionally declares one auto-delete queue

Transcript conventions: <scratch> stands for the session's scratch directory; 41533 and 54251 are the upstream container's local HTTP and AMQP ports ($AMQP in the one backgrounded command); ANSI colours and curl's progress meter are removed. Nothing else is edited.

The upstream container was started with the boot file loaded through load_definitions, as the chart does:

docker run -d --name syrf-p04-upstream-<random> --hostname proofnode \
  -p 127.0.0.1:<free>:15672 -p 127.0.0.1:<free>:5672 \
  -v <scratch>/proof/upstream-boot:/proof/boot:ro \
  -v <scratch>/proof/upstream-boot/90-proof.conf:/etc/rabbitmq/conf.d/90-proof.conf:ro \
  rabbitmq:3.13.6-management
# 90-proof.conf: load_definitions = /proof/boot/definitions.json

The Bitnami containers used the chart's own image, its rendered rabbitmq.conf mounted at /bitnami/rabbitmq/conf/rabbitmq.conf, its rendered environment, the definitions at /app/load_definition.json and a named volume at the PVC mount path /opt/bitnami/rabbitmq/.rabbitmq/mnesia. Differences from the cluster, none of which touch user handling: the Kubernetes peer-discovery, TLS-listener and Prometheus lines were dropped from the config, only rabbitmq_management was enabled (no delayed-exchange download), the node name was rabbit@localhost with short names, and the container did not run with a read-only root filesystem.

docker run -d --name syrf-p04-bitnami-<random>-b<n> --hostname localhost \
  -p 127.0.0.1:<free>:15672 \
  -v syrf-p04-bitnami-<random>-v<n>:/opt/bitnami/rabbitmq/.rabbitmq/mnesia \
  -v <scratch>/proof/bn-rabbitmq.conf.ldtrue:/bitnami/rabbitmq/conf/rabbitmq.conf:ro \
  -v <scratch>/proof/bn-app-b<n>:/app:ro \
  -e BITNAMI_DEBUG=false -e RABBITMQ_FORCE_BOOT=no \
  -e RABBITMQ_NODE_NAME=rabbit@localhost -e RABBITMQ_USE_LONGNAME=false \
  -e RABBITMQ_MNESIA_DIR=/opt/bitnami/rabbitmq/.rabbitmq/mnesia/rabbit@localhost \
  -e RABBITMQ_LDAP_ENABLE=no -e RABBITMQ_LOGS=- -e RABBITMQ_ULIMIT_NOFILES=65535 \
  -e RABBITMQ_ERL_COOKIE=local-proof-cookie-not-secret \
  -e RABBITMQ_LOAD_DEFINITIONS=yes -e RABBITMQ_DEFINITIONS_FILE=/app/load_definition.json \
  -e RABBITMQ_SECURE_PASSWORD=yes -e RABBITMQ_USERNAME=rabbit -e RABBITMQ_PASSWORD=test-rabbit-1 \
  -e RABBITMQ_PLUGINS=rabbitmq_management \
  docker.io/bitnamilegacy/rabbitmq:3.13.6-debian-12-r1
# B3/B3b: RABBITMQ_LOAD_DEFINITIONS=no, the loadDefinition=false config, no /app mount.

A. Live definitions import on `rabbitmq:3.13.6-management`¶

The container boots from v1-boot.json, which mirrors today's rendered definitions: rabbit (administrator, .* on /, syrf-production, syrf-staging and local), the two notifiers, the two PDF agents and the observer. Each later file is a jq edit of the previous one, shown in the transcript before it is posted.

A0. Baseline¶

The boot import created the six listed users. The beam process is PID "20".

$ docker image inspect rabbitmq:3.13.6-management --format '{{index .RepoDigests 0}}'
rabbitmq@sha256:b61a8a862766e4c5b526dd0cb0c382b3f006f25b21d9bf7e425ae6c7011ee87b

$ curl -sS -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/overview?columns=rabbitmq_version,erlang_version,product_version'; echo
{"product_version":"3.13.6","rabbitmq_version":"3.13.6","erlang_version":"26.2.5.2"}

$ docker inspect syrf-p04-upstream-3157a7bc --format 'StartedAt={{.State.StartedAt}} RestartCount={{.RestartCount}}'
StartedAt=2026-10-14T12:14:26.514784371Z RestartCount=0

$ docker exec -u rabbitmq syrf-p04-upstream-3157a7bc rabbitmqctl eval 'os:getpid().'
"20"

$ curl -sS -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/users?columns=name,tags&disable_stats=true' | jq -c '.[]'
{"name":"rabbit","tags":["administrator"]}
{"name":"syrf-broker-observer","tags":["monitoring"]}
{"name":"syrf-production-pdfagent","tags":[]}
{"name":"syrf-production-s3notifier","tags":[]}
{"name":"syrf-staging-pdfagent","tags":[]}
{"name":"syrf-staging-s3notifier","tags":[]}

$ curl -sS -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/users/syrf-staging-s3notifier/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"syrf-staging","configure":".*","write":".*","read":".*"}

A1. Live POST /api/definitions¶

Posted as rabbit. One request creates syrf-broker-provisioner and syrf-staging-api, changes the observer's password, narrows syrf-staging-s3notifier to one queue name and puts syrf-production-pdfagent on the Paused set. Every change is live at once; nothing restarted.

$ jq -c '[.users[] | {name,tags}]' <scratch>/proof/defs/v2-live-upsert.json
[{"name":"rabbit","tags":"administrator"},{"name":"syrf-staging-s3notifier","tags":""},{"name":"syrf-production-s3notifier","tags":""},{"name":"syrf-staging-pdfagent","tags":""},{"name":"syrf-production-pdfagent","tags":""},{"name":"syrf-broker-observer","tags":"monitoring"},{"name":"syrf-broker-provisioner","tags":"administrator"},{"name":"syrf-staging-api","tags":""}]

$ jq -c '.permissions[] | select(.user=="syrf-staging-s3notifier" or .user=="syrf-production-pdfagent" or .user=="syrf-staging-api")' <scratch>/proof/defs/v2-live-upsert.json
{"user":"syrf-staging-s3notifier","vhost":"syrf-staging","configure":"^SyRF\\.ProjectManagement\\.IBulkPdfUploadObjectReadyCommand$","write":"^SyRF\\.ProjectManagement\\.IBulkPdfUploadObjectReadyCommand$","read":"^SyRF\\.ProjectManagement\\.IBulkPdfUploadObjectReadyCommand$"}
{"user":"syrf-production-pdfagent","vhost":"syrf-production","configure":"^$","write":"^$","read":"^$"}
{"user":"syrf-staging-api","vhost":"syrf-staging","configure":".*","write":".*","read":".*"}

$ curl -sS -o /dev/null -w 'HTTP %{http_code}\n' -u rabbit:test-rabbit-1 -H content-type:application/json -X POST --data-binary @<scratch>/proof/defs/v2-live-upsert.json 'http://127.0.0.1:41533/api/definitions'
HTTP 204

$ curl -sS -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/users?columns=name,tags&disable_stats=true' | jq -c '.[]'
{"name":"rabbit","tags":["administrator"]}
{"name":"syrf-broker-observer","tags":["monitoring"]}
{"name":"syrf-broker-provisioner","tags":["administrator"]}
{"name":"syrf-production-pdfagent","tags":[]}
{"name":"syrf-production-s3notifier","tags":[]}
{"name":"syrf-staging-api","tags":[]}
{"name":"syrf-staging-pdfagent","tags":[]}
{"name":"syrf-staging-s3notifier","tags":[]}

$ curl -sS -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/users/syrf-staging-s3notifier/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"syrf-staging","configure":"^SyRF\\.ProjectManagement\\.IBulkPdfUploadObjectReadyCommand$","write":"^SyRF\\.ProjectManagement\\.IBulkPdfUploadObjectReadyCommand$","read":"^SyRF\\.ProjectManagement\\.IBulkPdfUploadObjectReadyCommand$"}

$ curl -sS -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/users/syrf-production-pdfagent/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"syrf-production","configure":"^$","write":"^$","read":"^$"}

$ curl -sS -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/users/syrf-staging-api/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"syrf-staging","configure":".*","write":".*","read":".*"}

$ curl -sS -o /dev/null -w 'observer old password: HTTP %{http_code}\n' -u syrf-broker-observer:test-observer-1 'http://127.0.0.1:41533/api/whoami'
observer old password: HTTP 401

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-observer:test-observer-2 'http://127.0.0.1:41533/api/whoami'
{"name":"syrf-broker-observer","tags":["monitoring"],"is_internal_user":true}  HTTP 200

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/whoami'
{"name":"syrf-broker-provisioner","tags":["administrator"],"is_internal_user":true}  HTTP 200

$ python3 -I <scratch>/scripts/amqp_probe.py 127.0.0.1 54251 syrf-staging-api test-stg-api-1 syrf-staging proof.api.q
auth user=syrf-staging-api: OK; vhost syrf-staging: OPEN
queue.declare proof.api.q: OK

$ python3 -I <scratch>/scripts/amqp_probe.py 127.0.0.1 54251 syrf-staging-s3notifier test-stg-s3n-1 syrf-staging proof.not.allowed
auth user=syrf-staging-s3notifier: OK; vhost syrf-staging: OPEN
queue.declare proof.not.allowed: REFUSED 403 ACCESS_REFUSED - configure access to queue 'proof.not.allowed' in vhost 'syrf-staging' refused for user 'syrf-staging-s3notifier'
(exit 4)

$ python3 -I <scratch>/scripts/amqp_probe.py 127.0.0.1 54251 syrf-production-pdfagent test-prd-pdf-1 syrf-production proof.not.allowed
auth user=syrf-production-pdfagent: OK; vhost syrf-production: OPEN
queue.declare proof.not.allowed: REFUSED 403 ACCESS_REFUSED - configure access to queue 'proof.not.allowed' in vhost 'syrf-production' refused for user 'syrf-production-pdfagent'
(exit 4)

$ docker inspect syrf-p04-upstream-3157a7bc --format 'StartedAt={{.State.StartedAt}} RestartCount={{.RestartCount}}'
StartedAt=2026-10-14T12:14:26.514784371Z RestartCount=0

$ docker exec -u rabbitmq syrf-p04-upstream-3157a7bc rabbitmqctl eval 'os:getpid().'
"20"

A2. A live import that omits users¶

From here on the requests run as the provisioner. The file leaves out syrf-production-pdfagent and syrf-staging-api and their permissions. Both users, and their grants, are still there afterwards.

$ jq -c '[.users[].name]' <scratch>/proof/defs/v3-omit.json
["rabbit","syrf-staging-s3notifier","syrf-production-s3notifier","syrf-staging-pdfagent","syrf-broker-observer","syrf-broker-provisioner"]

$ curl -sS -o /dev/null -w 'HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 -H content-type:application/json -X POST --data-binary @<scratch>/proof/defs/v3-omit.json 'http://127.0.0.1:41533/api/definitions'
HTTP 204

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users?columns=name,tags&disable_stats=true' | jq -c '.[]'
{"name":"rabbit","tags":["administrator"]}
{"name":"syrf-broker-observer","tags":["monitoring"]}
{"name":"syrf-broker-provisioner","tags":["administrator"]}
{"name":"syrf-production-pdfagent","tags":[]}
{"name":"syrf-production-s3notifier","tags":[]}
{"name":"syrf-staging-api","tags":[]}
{"name":"syrf-staging-pdfagent","tags":[]}
{"name":"syrf-staging-s3notifier","tags":[]}

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/syrf-production-pdfagent/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"syrf-production","configure":"^$","write":"^$","read":"^$"}

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/syrf-staging-api/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"syrf-staging","configure":".*","write":".*","read":".*"}

$ python3 -I <scratch>/scripts/amqp_probe.py 127.0.0.1 54251 syrf-staging-api test-stg-api-1 syrf-staging proof.api.q
auth user=syrf-staging-api: OK; vhost syrf-staging: OPEN
queue.declare proof.api.q: OK

A3. Demote rabbit and drop its permission entries¶

This is Phase 5.2 as the plan wrote it. The tag change takes effect at once (401 on the management API), but the four .* grants survive because nothing in the file mentions them, and rabbit can still declare a queue in syrf-production over AMQP.

$ jq -c '.users[] | select(.name=="rabbit") | {name,tags}' <scratch>/proof/defs/v4-demote.json; jq -c '[.permissions[] | select(.user=="rabbit")] | length' <scratch>/proof/defs/v4-demote.json
{"name":"rabbit","tags":""}
0

$ curl -sS -o /dev/null -w 'HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 -H content-type:application/json -X POST --data-binary @<scratch>/proof/defs/v4-demote.json 'http://127.0.0.1:41533/api/definitions'
HTTP 204

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/rabbit' | jq -c '{name,tags}'
{"name":"rabbit","tags":[]}

$ curl -sS -w '  HTTP %{http_code}\n' -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/whoami'
{"error":"not_authorised","reason":"Not management user"}  HTTP 401

$ curl -sS -w '  HTTP %{http_code}\n' -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/overview?columns=rabbitmq_version'
{"error":"not_authorised","reason":"Not management user"}  HTTP 401

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/rabbit/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"/","configure":".*","write":".*","read":".*"}
{"vhost":"local","configure":".*","write":".*","read":".*"}
{"vhost":"syrf-production","configure":".*","write":".*","read":".*"}
{"vhost":"syrf-staging","configure":".*","write":".*","read":".*"}

$ python3 -I <scratch>/scripts/amqp_probe.py 127.0.0.1 54251 rabbit test-rabbit-1 syrf-production proof.rabbit.q
auth user=rabbit: OK; vhost syrf-production: OPEN
queue.declare proof.rabbit.q: OK

A3b. Explicit ^$ entries¶

Writing ^$ for each vhost through the same import removes resource access live. The vhost itself still opens: in RabbitMQ a permission entry, whatever its patterns, is what grants vhost access.

$ jq -c '.permissions[] | select(.user=="rabbit")' <scratch>/proof/defs/v5-demote-explicit.json
{"user":"rabbit","vhost":"/","configure":"^$","write":"^$","read":"^$"}
{"user":"rabbit","vhost":"syrf-production","configure":"^$","write":"^$","read":"^$"}
{"user":"rabbit","vhost":"syrf-staging","configure":"^$","write":"^$","read":"^$"}
{"user":"rabbit","vhost":"local","configure":"^$","write":"^$","read":"^$"}

$ curl -sS -o /dev/null -w 'HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 -H content-type:application/json -X POST --data-binary @<scratch>/proof/defs/v5-demote-explicit.json 'http://127.0.0.1:41533/api/definitions'
HTTP 204

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/rabbit/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"/","configure":"^$","write":"^$","read":"^$"}
{"vhost":"local","configure":"^$","write":"^$","read":"^$"}
{"vhost":"syrf-production","configure":"^$","write":"^$","read":"^$"}
{"vhost":"syrf-staging","configure":"^$","write":"^$","read":"^$"}

$ python3 -I <scratch>/scripts/amqp_probe.py 127.0.0.1 54251 rabbit test-rabbit-1 syrf-production proof.rabbit.q
auth user=rabbit: OK; vhost syrf-production: OPEN
queue.declare proof.rabbit.q: REFUSED 403 ACCESS_REFUSED - configure access to queue 'proof.rabbit.q' in vhost 'syrf-production' refused for user 'rabbit'
(exit 4)

A3c. DELETE the permission entry¶

Deleting the entry is the only way to remove vhost access short of deleting the user.

$ curl -sS -o /dev/null -w 'HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 -X DELETE 'http://127.0.0.1:41533/api/permissions/syrf-production/rabbit'
HTTP 204

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/rabbit/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"/","configure":"^$","write":"^$","read":"^$"}
{"vhost":"local","configure":"^$","write":"^$","read":"^$"}
{"vhost":"syrf-staging","configure":"^$","write":"^$","read":"^$"}

$ python3 -I <scratch>/scripts/amqp_probe.py 127.0.0.1 54251 rabbit test-rabbit-1 syrf-production
auth user=rabbit: OK; vhost syrf-production: REFUSED 530 NOT_ALLOWED - access to vhost 'syrf-production' refused for user 'rabbit'
(exit 3)

A4. DELETE /api/users/rabbit¶

A second rabbit connection is held open in the background first. The delete closes it within the hold window with CONNECTION_FORCED, and later logins are refused on both HTTP and AMQP. The PID is still "20": nothing in A1 to A4 restarted the broker.

$ python3 -I <scratch>/scripts/amqp_probe.py 127.0.0.1 $AMQP rabbit test-rabbit-1 syrf-staging - 20 > held.txt &   # background

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/vhosts/syrf-staging/connections?columns=user,state' | jq -c '.[]'
{"state":"running","user":"rabbit"}

$ curl -sS -o /dev/null -w 'HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 -X DELETE 'http://127.0.0.1:41533/api/users/rabbit'
HTTP 204

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/rabbit'
{"error":"Object Not Found","reason":"Not Found"}  HTTP 404

$ curl -sS -w '  HTTP %{http_code}\n' -u rabbit:test-rabbit-1 'http://127.0.0.1:41533/api/whoami'
{"error":"not_authorized","reason":"Not_Authorized"}  HTTP 401

$ python3 -I <scratch>/scripts/amqp_probe.py 127.0.0.1 54251 rabbit test-rabbit-1 syrf-staging
auth user=rabbit: REFUSED (socket closed by server)
(exit 2)

$ cat <scratch>/proof/held.txt
auth user=rabbit: OK; vhost syrf-staging: OPEN
held connection: server closed it 320 CONNECTION_FORCED - user 'rabbit' is deleted

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/vhosts/syrf-staging/connections?columns=user,state' | jq -c '.[]'

$ docker inspect syrf-p04-upstream-3157a7bc --format 'StartedAt={{.State.StartedAt}} RestartCount={{.RestartCount}}'
StartedAt=2026-10-14T12:14:26.514784371Z RestartCount=0

$ docker exec -u rabbitmq syrf-p04-upstream-3157a7bc rabbitmqctl eval 'os:getpid().'
"20"

A5. Is an import all-or-nothing? (risk K2)¶

The file is valid JSON whose last permission names a vhost that exists neither in the file nor on the broker. The request fails with 400, yet the user listed earlier in the same file was created. Malformed JSON is rejected before anything is applied.

$ jq -c '.users[-1], .permissions[-1]' <scratch>/proof/defs/k2-partial.json
{"name":"syrf-k2-probe","password":"test-k2-1","tags":""}
{"user":"syrf-k2-probe","vhost":"vhost-that-does-not-exist","configure":".*","write":".*","read":".*"}

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 -H content-type:application/json -X POST --data-binary @<scratch>/proof/defs/k2-partial.json 'http://127.0.0.1:41533/api/definitions'
{"error":"bad_request","reason":"Please create virtual host \"vhost-that-does-not-exist\" prior to importing definitions."}  HTTP 400

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/syrf-k2-probe' | jq -c '{name,tags}' 2>/dev/null || true
{"name":"syrf-k2-probe","tags":[]}

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/syrf-k2-probe'
{"name":"syrf-k2-probe","password_hash":"Ywl+5ifhtyaf+kGlXw9Ed/8o3HVeHPeEdH50TJbm0UsvAGy1","hashing_algorithm":"rabbit_password_hashing_sha256","tags":[],"limits":{}}  HTTP 200

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/vhosts/vhost-that-does-not-exist'
{"error":"Object Not Found","reason":"Not Found"}  HTTP 404

$ cat <scratch>/proof/defs/k2-malformed.json; echo
{"users": [ {"name": "syrf-k2-malformed", "password": "x", "tags": ""} 

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 -H content-type:application/json -X POST --data-binary @<scratch>/proof/defs/k2-malformed.json 'http://127.0.0.1:41533/api/definitions'
{"error":"bad_request","reason":"failed_to_parse_json"}  HTTP 400

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/syrf-k2-malformed'
{"error":"Object Not Found","reason":"Not Found"}  HTTP 404

A6. Restart re-applies the boot file¶

The boot file is still v1-boot.json. After the restart (new PID "21"), rabbit is back as administrator with its four .* grants, and the notifier's narrowed grant is back to the file's .*. Users that only the live imports created (syrf-broker-provisioner, syrf-staging-api, syrf-k2-probe) survive the restart, because the boot import never deletes.

$ docker restart syrf-p04-upstream-3157a7bc >/dev/null && echo restarted
restarted

$ docker inspect syrf-p04-upstream-3157a7bc --format 'StartedAt={{.State.StartedAt}} RestartCount={{.RestartCount}}'
StartedAt=2026-10-14T12:14:49.070936426Z RestartCount=0

$ docker exec -u rabbitmq syrf-p04-upstream-3157a7bc rabbitmqctl eval 'os:getpid().'
"21"

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users?columns=name,tags&disable_stats=true' | jq -c '.[]'
{"name":"rabbit","tags":["administrator"]}
{"name":"syrf-broker-observer","tags":["monitoring"]}
{"name":"syrf-broker-provisioner","tags":["administrator"]}
{"name":"syrf-k2-probe","tags":[]}
{"name":"syrf-production-pdfagent","tags":[]}
{"name":"syrf-production-s3notifier","tags":[]}
{"name":"syrf-staging-api","tags":[]}
{"name":"syrf-staging-pdfagent","tags":[]}
{"name":"syrf-staging-s3notifier","tags":[]}

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/syrf-staging-s3notifier/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"syrf-staging","configure":".*","write":".*","read":".*"}

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/rabbit/permissions' | jq -c '.[] | {vhost,configure,write,read}'
{"vhost":"/","configure":".*","write":".*","read":".*"}
{"vhost":"local","configure":".*","write":".*","read":".*"}
{"vhost":"syrf-production","configure":".*","write":".*","read":".*"}
{"vhost":"syrf-staging","configure":".*","write":".*","read":".*"}

A6b. Boot file without rabbit, then restart¶

With rabbit gone from the boot file and deleted live, it stays deleted across a restart. This is the Phase 5.5 order: change the file first, then delete.

$ jq -c '[.users[].name]' <scratch>/proof/upstream-boot/definitions.json
["syrf-staging-s3notifier","syrf-production-s3notifier","syrf-staging-pdfagent","syrf-broker-observer","syrf-broker-provisioner"]

$ curl -sS -o /dev/null -w 'HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 -X DELETE 'http://127.0.0.1:41533/api/users/rabbit'
HTTP 204

$ docker restart syrf-p04-upstream-3157a7bc >/dev/null && echo restarted
restarted

$ curl -sS -w '  HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users/rabbit'
{"error":"Object Not Found","reason":"Not Found"}  HTTP 404

$ curl -sS -u syrf-broker-provisioner:test-provisioner-1 'http://127.0.0.1:41533/api/users?columns=name,tags&disable_stats=true' | jq -c '.[]'
{"name":"syrf-broker-observer","tags":["monitoring"]}
{"name":"syrf-broker-provisioner","tags":["administrator"]}
{"name":"syrf-k2-probe","tags":[]}
{"name":"syrf-production-pdfagent","tags":[]}
{"name":"syrf-production-s3notifier","tags":[]}
{"name":"syrf-staging-api","tags":[]}
{"name":"syrf-staging-pdfagent","tags":[]}
{"name":"syrf-staging-s3notifier","tags":[]}

H. Bitnami chart 14.6.6 render with the real values¶

Rendered with an isolated Helm repository configuration in the scratch directory, from the values on cluster-gitops main. The long base64 line that the diffs skip is the chart's rabbitmq.conf Secret; it is decoded and diffed separately.

H0. Inputs¶

The values file and chart version that Argo CD deploys.

$ helm version --short
v3.16.4+g7877b45

$ gh api 'repos/camaradesuk/cluster-gitops/commits?path=plugins/helm/rabbitmq/values.yaml&sha=main&per_page=1' --jq '.[0] | .sha[0:8] + " " + (.commit.message | split("\n")[0])'
bdb71e62 Merge pull request #1639 from camaradesuk/feat/remove-public-rabbitmq-management-ingress-2a0iw1

$ gh api 'repos/camaradesuk/cluster-gitops/contents/plugins/helm/rabbitmq/values.yaml?ref=main' --jq .content | base64 -d > values.yaml && grep -nE '^(auth|loadDefinition|ingress):|^  (username|existingPasswordSecret|enabled|existingSecret):' values.yaml
21:auth:
24:  username: rabbit
26:  existingPasswordSecret: rabbit-mq  # From extra-secrets chart
45:  enabled: true
66:  enabled: true
80:  enabled: true
95:  enabled: true
107:ingress:
108:  enabled: false
115:loadDefinition:
116:  enabled: true
117:  existingSecret: rabbitmq-definitions

$ gh api 'repos/camaradesuk/cluster-gitops/contents/plugins/helm/rabbitmq/config.yaml?ref=main' --jq .content | base64 -d | grep -E 'chart|version'
  repoURL: https://charts.bitnami.com/bitnami
  chart: rabbitmq
  version: "14.6.6"

$ helm repo add bitnami https://charts.bitnami.com/bitnami >/dev/null && echo added
added

H1. Probes with the live values¶

Both probes run rabbitmq-diagnostics, which authenticates with the Erlang cookie, not a broker user. RABBITMQ_PASSWORD comes from rabbit-mq through a non-optional secretKeyRef.

$ helm template rabbitmq bitnami/rabbitmq --version 14.6.6 -n rabbitmq -f values.yaml > ld-true.yaml && grep -nE 'helm.sh/chart' ld-true.yaml | head -1
13:    helm.sh/chart: rabbitmq-14.6.6

$ grep -nE -A10 '^          (livenessProbe|readinessProbe):' ld-true.yaml
418:          livenessProbe:
419-            failureThreshold: 6
420-            initialDelaySeconds: 120
421-            periodSeconds: 30
422-            successThreshold: 1
423-            timeoutSeconds: 20
424-            exec:
425-              command:
426-                - sh
427-                - -ec
428-                - rabbitmq-diagnostics -q ping
429:          readinessProbe:
430-            failureThreshold: 3
431-            initialDelaySeconds: 10
432-            periodSeconds: 30
433-            successThreshold: 1
434-            timeoutSeconds: 20
435-            exec:
436-              command:
437-                - sh
438-                - -ec
439-                - rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms

$ grep -nE -A1 'name: RABBITMQ_(LOAD_DEFINITIONS|USERNAME|SECURE_PASSWORD)$' ld-true.yaml
387:            - name: RABBITMQ_LOAD_DEFINITIONS
388-              value: "yes"
--
391:            - name: RABBITMQ_SECURE_PASSWORD
392-              value: "yes"
393:            - name: RABBITMQ_USERNAME
394-              value: "rabbit"

$ grep -nE -A4 'name: RABBITMQ_PASSWORD$' ld-true.yaml
395:            - name: RABBITMQ_PASSWORD
396-              valueFrom:
397-                secretKeyRef:
398-                  name: rabbit-mq
399-                  key: rabbitmq-password

H2. The same render with loadDefinition false¶

Only the probes, RABBITMQ_LOAD_DEFINITIONS, the definitions volume and the load_definitions line change. The probes become curl --user rabbit:$RABBITMQ_PASSWORD calls to the management API. Note that default_user = rabbit is in the config either way; RabbitMQ ignores it when definitions are loaded (B1).

$ helm template rabbitmq bitnami/rabbitmq --version 14.6.6 -n rabbitmq -f values.yaml --set loadDefinition.enabled=false > ld-false.yaml

$ grep -nE -A10 '^          (livenessProbe|readinessProbe):' ld-false.yaml
418:          livenessProbe:
419-            failureThreshold: 6
420-            initialDelaySeconds: 120
421-            periodSeconds: 30
422-            successThreshold: 1
423-            timeoutSeconds: 20
424-            exec:
425-              command:
426-                - sh
427-                - -ec
428-                - curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts
429:          readinessProbe:
430-            failureThreshold: 3
431-            initialDelaySeconds: 10
432-            periodSeconds: 30
433-            successThreshold: 1
434-            timeoutSeconds: 20
435-            exec:
436-              command:
437-                - sh
438-                - -ec
439-                - curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms

$ diff ld-true.yaml ld-false.yaml | grep -vE '^[<>]     [A-Za-z0-9+/=]{100,}$'
105c105
---
257c257
<         checksum/config: 492403e100cf0e058f41dde65e9ba1b4efebaa86a10084721ae404ee779d00ed
---
>         checksum/config: 1d84c94948f8e8422bd140f46942a153cd1dcaf6a82ac3aaa6904ad94814dbc2
388c388
<               value: "yes"
---
>               value: "no"
428c428
<                 - rabbitmq-diagnostics -q ping
---
>                 - curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts
439c439
<                 - rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms
---
>                 - curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms
472,474d471
<             - name: load-definition-volume
<               mountPath: /app
<               readOnly: true
495,497d491
<         - name: load-definition-volume
<           secret:
<             secretName: "rabbitmq-definitions"

$ conf() { awk '/^  rabbitmq.conf: \|-/{getline; print; exit}' "$1" | tr -d ' ' | base64 -d; }; conf ld-true.yaml > conf-true; conf ld-false.yaml > conf-false; grep -nE '^(default_user|load_definitions|loopback_users)' conf-true; echo ---; diff conf-true conf-false
2:default_user = rabbit
17:load_definitions = /app/load_definition.json
21:loopback_users.rabbit = false
---
15,17d14
< cluster_formation.target_cluster_size_hint = 1
< 
< load_definitions = /app/load_definition.json
(exit 1)

H3. Phase 5.4 as planned¶

Switching the chart's user to break-glass changes the pod template (env and checksum/config), so the StatefulSet rolls the broker pod.

$ helm template rabbitmq bitnami/rabbitmq --version 14.6.6 -n rabbitmq -f values.yaml --set auth.username=syrf-broker-breakglass --set auth.existingPasswordSecret=rabbitmq-breakglass > breakglass.yaml

$ diff ld-true.yaml breakglass.yaml | grep -vE '^[<>]     [A-Za-z0-9+/=]{100,}$'
88c88
<   - name: rabbit-mq
---
>   - name: rabbitmq-breakglass
105c105
---
257c257
<         checksum/config: 492403e100cf0e058f41dde65e9ba1b4efebaa86a10084721ae404ee779d00ed
---
>         checksum/config: f050ad8060294836fa15d7e62e98a99e8cdb28dabe3188fcc62a93d91d4f2870
394c394
<               value: "rabbit"
---
>               value: "syrf-broker-breakglass"
398c398
<                   name: rabbit-mq
---
>                   name: rabbitmq-breakglass

$ conf() { awk '/^  rabbitmq.conf: \|-/{getline; print; exit}' "$1" | tr -d ' ' | base64 -d; }; conf breakglass.yaml > conf-bg; diff conf-true conf-bg
2c2
< default_user = rabbit
---
> default_user = syrf-broker-breakglass
21c21
< loopback_users.rabbit = false
---
> loopback_users.syrf-broker-breakglass = false
(exit 1)

B. Bitnami start-up behaviour in Docker¶

Each case runs the rendered probe commands with docker exec, so a non-zero exit is what the kubelet would see. The loadDefinition: false probes are run in every case to show what they would do.

B0. Settings under test¶

The trimmed chart config used for the loadDefinition: true cases.

$ docker image inspect docker.io/bitnamilegacy/rabbitmq:3.13.6-debian-12-r1 --format '{{index .RepoDigests 0}}'
bitnamilegacy/rabbitmq@sha256:c6e65c758507a458c7f99e06cde560e0da21c848cdd0473243d8ef9504697246

$ cat <scratch>/proof/bn-rabbitmq.conf.ldtrue
## Username and password
default_user = rabbit
## Clustering
##
cluster_name = rabbitmq
cluster_partition_handling = autoheal

cluster_formation.target_cluster_size_hint = 1

load_definitions = /app/load_definition.json
# queue master locator
queue_master_locator = min-masters
# enable loopback user
loopback_users.rabbit = false
#default_vhost = rabbitmq-vhost
#disk_free_limit.absolute = 50MB
## Prometheus metrics
##
## Memory Threshold
##
total_memory_available_override_value = 1073741824
vm_memory_high_watermark.relative = 0.6
## TCP Listen Options
##
tcp_listen_options.backlog = 128
tcp_listen_options.nodelay = true
tcp_listen_options.linger.on      = true
tcp_listen_options.linger.timeout = 0
tcp_listen_options.keepalive = false

B1. Empty PVC, definitions without rabbit¶

This is a lost PVC after Phase 5.5. RabbitMQ skips the default user because definitions are configured, and the Bitnami script does not add auth.username because the file has a "users" key. rabbit does not exist; the live probes pass; the curl probes would fail with 401.

$ jq -c '[.users[] | {name,tags}]' <scratch>/proof/bn-app-b1/load_definition.json
[{"name":"syrf-broker-provisioner","tags":"administrator"},{"name":"syrf-broker-observer","tags":"monitoring"}]

$ docker logs syrf-p04-bitnami-b1bb846e-b1 2>&1 | grep -E 'rabbitmq (\[|.*INFO)|Persisted|Initializing|definitions|default (user|virtual host)|Will not seed' | grep -v -E 'Stopping|stopped' | cut -c1-220
rabbitmq 12:19:24.87 INFO  ==> 
rabbitmq 12:19:24.87 INFO  ==> Welcome to the Bitnami rabbitmq container
rabbitmq 12:19:24.88 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
rabbitmq 12:19:24.88 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
rabbitmq 12:19:24.88 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features,
rabbitmq 12:19:24.89 INFO  ==> 
rabbitmq 12:19:24.90 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq 12:19:24.93 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq 12:19:24.96 INFO  ==> Initializing RabbitMQ...
rabbitmq 12:19:25.03 INFO  ==> Starting RabbitMQ in background...
rabbitmq 12:19:33.47 INFO  ==> No custom scripts in /docker-entrypoint-initdb.d
rabbitmq 12:19:36.39 INFO  ==> ** RabbitMQ setup finished! **
rabbitmq 12:19:36.43 INFO  ==> Setting file description limit to 65535
rabbitmq 12:19:36.43 INFO  ==> ** Starting RabbitMQ **
2026-10-14 12:19:41.275182+00:00 [info] <0.254.0> Running boot step rabbit_definitions_hashing defined by app rabbit
2026-10-14 12:19:41.348081+00:00 [info] <0.254.0> Will not seed default virtual host and user: have definitions to load...
2026-10-14 12:19:41.354192+00:00 [info] <0.254.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2026-10-14 12:19:44.110641+00:00 [info] <0.686.0> Applying definitions from regular file at /app/load_definition.json
2026-10-14 12:19:44.110958+00:00 [info] <0.686.0> Applying definitions from file at '/app/load_definition.json'
2026-10-14 12:19:44.110993+00:00 [info] <0.686.0> Asked to import definitions. Acting user: rmq-internal

$ docker exec syrf-p04-bitnami-b1bb846e-b1 rabbitmqctl -q list_users
user    tags
syrf-broker-provisioner [administrator]
syrf-broker-observer    [monitoring]

# liveness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b1 sh -ec 'rabbitmq-diagnostics -q ping'; printf '\nexit=%s\n' $?
Ping succeeded

exit=0

# readiness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b1 sh -ec 'rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms'; printf '\nexit=%s\n' $?
RabbitMQ on node rabbit@localhost is fully booted and running
Node rabbit@localhost reported no local alarms

exit=0

# liveness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b1 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

# readiness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b1 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

B1b. Restart with persisted data¶

The Bitnami script takes the persisted-data branch and touches no user.

$ docker logs syrf-p04-bitnami-b1bb846e-b1 2>&1 | grep -E 'Persisted data detected|Initializing RabbitMQ' | cut -c1-200
rabbitmq 12:19:24.96 INFO  ==> Initializing RabbitMQ...
rabbitmq 12:19:51.53 INFO  ==> Initializing RabbitMQ...
rabbitmq 12:19:51.57 INFO  ==> Persisted data detected. Restoring...

$ docker exec syrf-p04-bitnami-b1bb846e-b1 rabbitmqctl -q list_users
user    tags
syrf-broker-provisioner [administrator]
syrf-broker-observer    [monitoring]

# liveness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b1 sh -ec 'rabbitmq-diagnostics -q ping'; printf '\nexit=%s\n' $?
Ping succeeded

exit=0

# readiness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b1 sh -ec 'rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms'; printf '\nexit=%s\n' $?
RabbitMQ on node rabbit@localhost is fully booted and running
Node rabbit@localhost reported no local alarms

exit=0

# liveness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b1 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

# readiness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b1 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

B2. rabbit demoted, then deleted, across restarts¶

rabbit starts as administrator. The definitions are changed to tags "" and the pod restarted: rabbit stays demoted (the chart does not re-tag it). Then rabbit is removed from the file, deleted live and the pod restarted: it stays deleted. The live probes pass throughout.

$ docker exec syrf-p04-bitnami-b1bb846e-b2 rabbitmqctl -q list_users
user    tags
syrf-broker-provisioner [administrator]
syrf-broker-observer    [monitoring]
rabbit  [administrator]

# liveness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'rabbitmq-diagnostics -q ping'; printf '\nexit=%s\n' $?
Ping succeeded

exit=0

# readiness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms'; printf '\nexit=%s\n' $?
RabbitMQ on node rabbit@localhost is fully booted and running
Node rabbit@localhost reported no local alarms

exit=0

# liveness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts'; printf '\nexit=%s\n' $?
{"status":"ok"}
exit=0

# readiness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms'; printf '\nexit=%s\n' $?
{"status":"ok"}
exit=0

$ jq -c '[.users[] | {name,tags}]' <scratch>/proof/bn-app-b2/load_definition.json
[{"name":"rabbit","tags":""},{"name":"syrf-broker-provisioner","tags":"administrator"},{"name":"syrf-broker-observer","tags":"monitoring"}]

$ docker logs syrf-p04-bitnami-b1bb846e-b2 2>&1 | grep -E 'Persisted data detected' | cut -c1-200
rabbitmq 12:20:30.55 INFO  ==> Persisted data detected. Restoring...

$ docker exec syrf-p04-bitnami-b1bb846e-b2 rabbitmqctl -q list_users
user    tags
syrf-broker-provisioner [administrator]
syrf-broker-observer    [monitoring]
rabbit  []

# liveness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'rabbitmq-diagnostics -q ping'; printf '\nexit=%s\n' $?
Ping succeeded

exit=0

# readiness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms'; printf '\nexit=%s\n' $?
RabbitMQ on node rabbit@localhost is fully booted and running
Node rabbit@localhost reported no local alarms

exit=0

# liveness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

# readiness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

$ jq -c '[.users[] | {name,tags}]' <scratch>/proof/bn-app-b2/load_definition.json
[{"name":"syrf-broker-provisioner","tags":"administrator"},{"name":"syrf-broker-observer","tags":"monitoring"}]

$ curl -sS -o /dev/null -w 'DELETE /api/users/rabbit: HTTP %{http_code}\n' -u syrf-broker-provisioner:test-provisioner-1 -X DELETE http://127.0.0.1:58163/api/users/rabbit
DELETE /api/users/rabbit: HTTP 204

$ docker exec syrf-p04-bitnami-b1bb846e-b2 rabbitmqctl -q list_users
user    tags
syrf-broker-provisioner [administrator]
syrf-broker-observer    [monitoring]

# liveness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'rabbitmq-diagnostics -q ping'; printf '\nexit=%s\n' $?
Ping succeeded

exit=0

# readiness as rendered (loadDefinition true)
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms'; printf '\nexit=%s\n' $?
RabbitMQ on node rabbit@localhost is fully booted and running
Node rabbit@localhost reported no local alarms

exit=0

# liveness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

# readiness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b2 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

B3. Counterfactual: loadDefinition false¶

On an empty PVC the chart creates rabbit as administrator (default_user plus rabbitmq_change_password). Once demoted, both curl probes fail with 401, and a restart does not repair it, so the pod would stay unready and the liveness probe would restart it repeatedly. This is why loadDefinition must stay enabled while auth.username names a demoted or deleted user.

$ diff <scratch>/proof/bn-rabbitmq.conf.ldtrue <scratch>/proof/bn-rabbitmq.conf.ldfalse
8,10d7
< cluster_formation.target_cluster_size_hint = 1
< 
< load_definitions = /app/load_definition.json
(exit 1)

$ docker exec syrf-p04-bitnami-b1bb846e-b3 rabbitmqctl -q list_users
user    tags
rabbit  [administrator]

# liveness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b3 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts'; printf '\nexit=%s\n' $?
{"status":"ok"}
100    15  100    15    0     0   7970      0 --:--:-- --:--:-- --:--:-- 15000

exit=0

# readiness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b3 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms'; printf '\nexit=%s\n' $?
{"status":"ok"}
exit=0

$ docker exec syrf-p04-bitnami-b1bb846e-b3 rabbitmqctl -q set_user_tags rabbit

$ docker exec syrf-p04-bitnami-b1bb846e-b3 rabbitmqctl -q list_users
user    tags
rabbit  []

# liveness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b3 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

# readiness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b3 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

$ docker logs syrf-p04-bitnami-b1bb846e-b3 2>&1 | grep -E 'Persisted data detected' | cut -c1-200
rabbitmq 12:21:24.51 INFO  ==> Persisted data detected. Restoring...

$ docker exec syrf-p04-bitnami-b1bb846e-b3 rabbitmqctl -q list_users
user    tags
rabbit  []

# liveness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b3 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

# readiness as rendered with loadDefinition false
$ docker exec syrf-p04-bitnami-b1bb846e-b3 sh -ec 'curl -f --user rabbit:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms'; printf '\nexit=%s\n' $?
curl: (22) The requested URL returned error: 401

exit=22

B3b. loadDefinition false, empty PVC¶

A second fresh start, for the record.

$ docker logs syrf-p04-bitnami-b1bb846e-b4 2>&1 | grep -E 'Initializing RabbitMQ|Persisted|default user|Creating user|Adding user|Changing password|password' | cut -c1-200
rabbitmq 12:21:35.78 INFO  ==> Initializing RabbitMQ...

$ docker exec syrf-p04-bitnami-b1bb846e-b4 rabbitmqctl -q list_users
user    tags
rabbit  [administrator]

Bitnami start-up source¶

The image's rabbitmq_initialize. With the chart's config mounted, line 804 skips the image's own config generation. Lines 842 to 849 are the only places that create or change auth.username, and they run only on an empty data directory.

$ docker run --rm --entrypoint sed docker.io/bitnamilegacy/rabbitmq:3.13.6-debian-12-r1 -n '800,805p;832,849p' /opt/bitnami/scripts/librabbitmq.sh
800:     # Check for mounted configuration files
801:     if ! is_dir_empty "$RABBITMQ_MOUNTED_CONF_DIR"; then
802:         cp -Lr "$RABBITMQ_MOUNTED_CONF_DIR"/* "$RABBITMQ_CONF_DIR"
803:     fi
804:     [[ ! -f "$RABBITMQ_CONF_FILE" ]] && rabbitmq_create_config_file
805:     rabbitmq_is_ssl_enabled && rabbitmq_create_combined_ssl_file
832:     # ref: https://github.com/bitnami/bitnami-docker-rabbitmq/pull/184
833:     if ! is_mounted_dir_empty "$(realpath "$RABBITMQ_DATA_DIR")"; then
834:         info "Persisted data detected. Restoring..."
835:         if is_boolean_yes "$RABBITMQ_FORCE_BOOT" && ! is_dir_empty "${RABBITMQ_DATA_DIR}/${RABBITMQ_NODE_NAME}"; then
836:             # ref: https://www.rabbitmq.com/rabbitmqctl.8.html#force_boot
837:             warn "Forcing node to start..."
838:             debug_execute "${RABBITMQ_BIN_DIR}/rabbitmqctl" force_boot
839:         fi
840:     else
841:         ! is_rabbitmq_running && rabbitmq_start_bg
842:         if is_boolean_yes "$RABBITMQ_LOAD_DEFINITIONS"; then
843:             if ! grep -q '"users"' "$RABBITMQ_DEFINITIONS_FILE"; then
844:                 debug_execute "${RABBITMQ_BIN_DIR}/rabbitmqctl" add_user "$RABBITMQ_USERNAME" "$RABBITMQ_PASSWORD"
845:                 debug_execute "${RABBITMQ_BIN_DIR}/rabbitmqctl" set_user_tags "$RABBITMQ_USERNAME" administrator
846:             fi
847:         elif is_boolean_yes "$RABBITMQ_SECURE_PASSWORD"; then
848:             rabbitmq_change_password "$RABBITMQ_USERNAME" "$RABBITMQ_PASSWORD"
849:         fi

Appendix: AMQP probe¶

Run as python3 -I amqp_probe.py HOST PORT USER PASSWORD VHOST [QUEUE|-] [HOLD_SECONDS].

#!/usr/bin/env python3
"""Minimal AMQP 0-9-1 probe (stdlib only) for local proofs.

Usage: amqp_probe.py HOST PORT USER PASSWORD VHOST [QUEUE] [HOLD_SECONDS]

Prints one line per step: auth/vhost open result, optional queue.declare result,
and (with HOLD_SECONDS) whether the server closed the connection while held open.
"""
import socket
import struct
import sys
import time


def shortstr(s):
    b = s.encode()
    return struct.pack("B", len(b)) + b


def longstr(b):
    return struct.pack(">I", len(b)) + b


def frame(ftype, channel, payload):
    return struct.pack(">BHI", ftype, channel, len(payload)) + payload + b"\xce"


def method(channel, cls, mid, args):
    return frame(1, channel, struct.pack(">HH", cls, mid) + args)


def read_exact(sock, n):
    buf = b""
    while len(buf) < n:
        chunk = sock.recv(n - len(buf))
        if not chunk:
            raise EOFError("socket closed by server")
        buf += chunk
    return buf


def read_frame(sock):
    ftype, channel, size = struct.unpack(">BHI", read_exact(sock, 7))
    payload = read_exact(sock, size)
    read_exact(sock, 1)
    return ftype, channel, payload


def read_method(sock):
    while True:
        ftype, channel, payload = read_frame(sock)
        if ftype == 1:
            cls, mid = struct.unpack(">HH", payload[:4])
            return channel, cls, mid, payload[4:]


def close_reason(args):
    code = struct.unpack(">H", args[:2])[0]
    n = args[2]
    return code, args[3:3 + n].decode(errors="replace")


def main():
    host, port, user, password, vhost = sys.argv[1:6]
    queue = sys.argv[6] if len(sys.argv) > 6 and sys.argv[6] != "-" else None
    hold = int(sys.argv[7]) if len(sys.argv) > 7 else 0
    sock = socket.create_connection((host, int(port)), timeout=10)
    sock.sendall(b"AMQP\x00\x00\x09\x01")
    _, cls, mid, _ = read_method(sock)
    assert (cls, mid) == (10, 10), (cls, mid)
    response = b"\x00" + user.encode() + b"\x00" + password.encode()
    sock.sendall(method(0, 10, 11, struct.pack(">I", 0) + shortstr("PLAIN") + longstr(response) + shortstr("en_US")))
    try:
        _, cls, mid, args = read_method(sock)
    except EOFError as exc:
        print(f"auth user={user}: REFUSED ({exc})")
        return 2
    if (cls, mid) == (10, 50):
        code, text = close_reason(args)
        print(f"auth user={user}: REFUSED {code} {text}")
        return 2
    channel_max, frame_max, _hb = struct.unpack(">HIH", args[:8])
    sock.sendall(method(0, 10, 31, struct.pack(">HIH", channel_max, frame_max, 0)))
    sock.sendall(method(0, 10, 40, shortstr(vhost) + shortstr("") + b"\x00"))
    try:
        _, cls, mid, args = read_method(sock)
    except EOFError as exc:
        print(f"auth user={user}: OK; vhost {vhost}: REFUSED ({exc})")
        return 3
    if (cls, mid) == (10, 50):
        code, text = close_reason(args)
        print(f"auth user={user}: OK; vhost {vhost}: REFUSED {code} {text}")
        return 3
    print(f"auth user={user}: OK; vhost {vhost}: OPEN")
    rc = 0
    if queue:
        sock.sendall(method(1, 20, 10, shortstr("")))
        _, cls, mid, _ = read_method(sock)
        assert (cls, mid) == (20, 11), (cls, mid)
        # queue.declare: reserved short, queue name, bits (auto-delete), empty arguments table
        sock.sendall(method(1, 50, 10, struct.pack(">H", 0) + shortstr(queue) + b"\x08" + struct.pack(">I", 0)))
        _, cls, mid, args = read_method(sock)
        if (cls, mid) == (50, 11):
            print(f"queue.declare {queue}: OK")
        else:
            code, text = close_reason(args)
            print(f"queue.declare {queue}: REFUSED {code} {text}")
            rc = 4
    if hold:
        sock.settimeout(1)
        deadline = time.time() + hold
        while time.time() < deadline:
            try:
                ftype, channel, payload = read_frame(sock)
            except socket.timeout:
                continue
            except EOFError as exc:
                print(f"held connection: server dropped it ({exc})")
                return 5
            if ftype == 1 and struct.unpack(">HH", payload[:4]) == (10, 50):
                code, text = close_reason(payload[4:])
                print(f"held connection: server closed it {code} {text}")
                return 5
        print(f"held connection: still open after {hold}s")
    sock.close()
    return rc


if __name__ == "__main__":
    sys.exit(main())