Enable versioning on the production upload bucket

This is the plan for turning on S3 versioning on the production uploads bucket syrfapp-uploads, with its rollback. Nothing here has been applied. Every production step is a deliberate cluster-gitops change that Chris approves and merges. The diffs below are drafts to copy, not changes that anyone has made.

Owner decision (Chris, 2026-10-14): fix the notifier now so that main is promotable, and prepare versioning as a separate, deliberate step. A rollback must be planned, and the cleanup rules must ship in the same deployment that first enables versioning. No separate cleanup job is needed; see What ships in the same deployment.

Tracking: syrf#3669 (production notifier and versioning), syrf#3923 (staging has no noncurrent expiry).

Current state (2026-10-14)

• syrfapp-uploads has never had versioning enabled. get-bucket-versioning returns {"Status":"Suspended"}, because the chart renders bucket.versioning: false as Suspended (cluster-gitops syrf/environments/production/s3-notifier/values.yaml). Every object has the version id null.
• Production runs a pre-v1.19 notifier package (CodeSha256 pbacxtlOf…). It ignores versionId and reads each object by key with s3:GetObject. The production role has s3:GetObject only: s3:GetObjectVersion is an implicit deny, because production is pinned at chart s3-notifier-v1.18.0 (syrf#3669).
• Notifier builds from v1.19.0 up to the last release before #4173 (v1.45.0 at the time of writing) throw on every event without a real versionId. They must never run against this bucket.
• syrf#4173 makes the notifier accept all three bucket states (see the notifier README). A real version id is pinned. A missing id or the literal "null" falls back to a by-key read, as the running package does. Bulk PDF without a real version is refused.
• Production Bulk PDF is off: featureFlags.bulkPdfUpload is set only in the staging values, and the chart refuses durableCapture for production.
• The production Bulk PDF flag must stay off until versioning (step 2) is verified. A Bulk PDF upload into a Suspended bucket is refused silently, with only an S3NotifierBulkPdfVersionRequired log line. No object-ready command is sent, so the web upload never completes.

What enabling versioning does

AWS sources: S3 Versioning, versioning-suspended buckets, event message structure, lifecycle elements, setting lifecycle configuration, enabling versioning.

• New writes get a unique version id. An overwrite keeps the previous bytes as a noncurrent version. A plain delete adds a delete marker and keeps the object as a noncurrent version.
• Existing objects do not change. They keep the version id null until they are next overwritten or deleted.
• Events carry the version. The versionId field is "the object version if the bucket is versioning-enabled; null or not present if the bucket isn't versioning-enabled". Only an Enabled bucket gives the notifier something to pin. The real-AWS rehearsal saw the field left out in every state that is not Enabled, never sent as "null".
• Every version is billed in full. "Each version of an object is the entire object; it is not just a diff from the previous version."
• Expiry rules change meaning. On a versioned bucket, an Expiration rule only adds a delete marker and keeps the object as noncurrent. AWS: "If you have an object expiration lifecycle configuration in your unversioned bucket and you want to maintain the same permanent delete behavior when you enable versioning, you must add a noncurrent expiration configuration." Production's expire-bulk-staging (7 days) is such a rule.
• First enablement takes up to 15 minutes to propagate. "During this time, GET requests for objects created or updated after enabling versioning may result in HTTP 404 NoSuchKey errors." A notifier HEAD that gets a 404 throws, and Lambda retries the event asynchronously.

The undo story

• There is no way back to "never versioned". "After you version-enable a bucket, it can never return to an unversioned state. But you can suspend versioning on that bucket."
• Suspended behaves like unversioned for new writes. S3 "automatically adds a null version ID to every subsequent object stored", and a write to a key that already has a null version replaces that version in place. A plain delete removes only the null version and adds a null delete marker.
• Suspending deletes nothing. "When you suspend versioning, existing objects in your bucket do not change." Every noncurrent version and delete marker created while Enabled stays, and stays billed, until something deletes it.
• Lifecycle still works on a Suspended bucket. In the AWS behaviour table, NoncurrentVersionExpiration on a versioning-suspended bucket is "Same behavior as a versioning-enabled bucket". An expired object delete marker (a delete marker with no versions left behind it) is removed by an ExpiredObjectDeleteMarker rule.

So rollback means: set Suspended, keep every lifecycle rule, and let expire-noncurrent-versions and remove-expired-delete-markers drain what versioning retained. An optional one-off purge (below) can drain it sooner. The end state is a Suspended bucket, which is exactly what production has today.

Lifecycle is asynchronous. Noncurrent days are counted "from when the version of the object becomes noncurrent", then rounded up to the next midnight UTC. AWS "might not actually delete these objects until days or even weeks later", but "changes in billing are applied when the object becomes eligible for the lifecycle action, even if the action isn't complete". Rule changes take "a few minutes" to propagate. In the real-AWS rehearsal, a 1-day noncurrent rule removed versions about 2 days after they became noncurrent, and the expired delete marker went one daily run later.

Cost

Read-only CloudWatch measurements from 2026-10-02 (cluster-gitops#1505):

Measure	Value
BucketSizeBytes	33.85 GB (about $0.78 a month at $0.023/GB-month, eu-west-1 Standard)
NumberOfObjects	8,511
Net growth	about 0.27 GB a month; the bucket never shrank on any of 180 days
Expected extra cost, 30-day noncurrent window	under 0.3 GB retained, under $0.01 a month
Worst case: whole bucket deleted by accident	33.85 GB kept for 30 days, about $0.78

Versioning adds no request charges. Re-measure both metrics the day before enabling (acceptance criterion 4); the estimate is from 2026-10-02.

Choosing N, the noncurrent retention

Proposal: N = 30 days for the whole bucket, and 1 day under Projects/_bulk-staging/. Chris confirmed N = 30 on 2026-10-02 (syrf#3669). This plan keeps that value. The reasoning against the Bulk PDF overwrite-protection need:

1. Integrity does not depend on N. Bulk PDF's overwrite protection means never processing different bytes from the ones the event named. The notifier sends the exact version id, Project Management stores it in the release record, and the agent revalidates that exact version before it downloads. If the version has been expired, the exact-version read fails and the job fails closed. N only decides how long an overwritten or deleted version stays available.
2. Legitimate Bulk PDF writes never overwrite. Staging keys are unique per upload, Projects/_bulk-staging/{projectId}/{uploadId}.zip. A noncurrent Bulk PDF version comes from only two sources. One is a ZIP deleted after processing, or expired by expire-bulk-staging after 7 days; nobody needs that version again. The other is an overwrite by a bug or a replay, which is exactly the case versioning protects.
3. One day still covers normal processing. With a 1-day rule, an overwritten original stays readable for 24 to 48 hours after the overwrite, because of the midnight rounding. A healthy agent claims within minutes. Staging's durable-capture terminal retention is PT24H. The current-object rule caps any Bulk PDF object at 7 days anyway. A longer window under _bulk-staging/ would mostly keep processed ZIPs, which can be large, for no benefit.
4. 30 days elsewhere is the recovery window. Search libraries, bulk study update files and study PDFs that a user deletes or overwrites by mistake stay recoverable for 30 days. That is far longer than any notifier or job retry, and it costs under $0.01 a month at today's churn.
5. Revisit before production Bulk PDF goes live. If measured p99 claim latency approaches 24 hours, raise the _bulk-staging/ value to 7. If durable capture is ever enabled in production, ADR-015 forbids age-based expiry under the reserved prefix. Remove expire-bulk-staging and its noncurrent rule then, as staging's values already did.

When several rules match a version, S3 applies permanent deletion as soon as any rule makes the version eligible. So the 1-day rule wins under _bulk-staging/, and the 30-day rule governs everything else.

What ships in the same deployment

One production values commit enables versioning. It must also carry the rules below. The shape comes from the held cluster-gitops#1505. Staging's values file does not carry it: staging is versioned with only an incomplete-multipart rule and no noncurrent expiry (syrf#3923), so do not copy staging.

Item	Shape	Why it must ship together
Incomplete Multipart Cleanup	unchanged, kept verbatim	ACK lifecycle rules are whole-desired-state: a rule left out is deleted
expire-bulk-staging	gains noncurrentVersionExpiration.noncurrentDays: 1	without it, expiry only hides orphaned ZIPs behind delete markers and they are billed forever
expire-noncurrent-versions	new, noncurrentVersionExpiration.noncurrentDays: 30, prefix ""	bounds every retained overwrite and delete to N days
remove-expired-delete-markers	new, expiration.expiredObjectDeleteMarker: true, prefix ""	removes delete markers with nothing behind them. It is a separate rule because S3 rejects days and expiredObjectDeleteMarker in one Expiration; both rehearsals confirmed this, real S3 with MalformedXML

Cleanup job: none needed. The lifecycle rules are the cleanup. S3 evaluates them daily. ACK renders them from the values file, so they deploy and roll back with the same commit as versioning. They keep working after a rollback to Suspended. A one-off purge is only an optional accelerator after a rollback (see Rollback).

Cost monitoring uses the free daily AWS/S3 storage metrics. BucketSizeBytes counts "both current and noncurrent objects". NumberOfObjects includes "current and noncurrent objects, delete markers, and the total number of parts for all incomplete multipart uploads" (metrics reference). Read them before enabling and at D+1, D+7 and D+35:

for metric in BucketSizeBytes:StandardStorage NumberOfObjects:AllStorageTypes; do
  aws cloudwatch get-metric-statistics --region eu-west-1 --namespace AWS/S3 \
    --metric-name "${metric%%:*}" --statistics Average --period 86400 \
    --start-time "$(date -u -d '-14 days' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
    --dimensions Name=BucketName,Value=syrfapp-uploads Name=StorageType,Value="${metric##*:}" \
    --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Average]' --output text
done
# Noncurrent versions and delete markers right now (8.5k objects, a handful of list calls):
aws s3api list-object-versions --bucket syrfapp-uploads --output json \
  | jq '{noncurrent: [.Versions[]? | select(.IsLatest | not)] | length,
         noncurrentBytes: ([.Versions[]? | select(.IsLatest | not) | .Size] | add // 0),
         deleteMarkers: [.DeleteMarkers[]?] | length}'

Investigate if BucketSizeBytes grows more than 2 GB above the pre-enable trend line by D+35. That is several times the expected noncurrent volume. A CloudWatch alarm on these metrics is a follow-up (below), not part of the enabling commit, because cluster-gitops has no established declarative owner for CloudWatch alarms yet.

Ordering against notifier code versions

Production notifier code	Bucket Suspended (today)	Bucket Enabled
Pre-v1.19 package pbacxtlOf… (running now)	Works: by-key reads	Works: ignores versionId
v1.19.0 up to the last release before #4173 (v1.45.0 at the time of writing)	Every event throws. Never deploy.	Works only if the role has s3:GetObjectVersion
First release containing #4173, and later	Works: by-key reads; Bulk PDF refused	Works: version-pinned reads; needs s3:GetObjectVersion

Two constraints follow:

• The notifier must be on the #4173 fix before or at the same time as versioning is enabled. Otherwise a later rollback to Suspended becomes an outage. Doing the code step first keeps every rollback safe.
• s3:GetObjectVersion must reach the production role no later than versioning. With Enabled, events carry real ids, and the fixed code issues version-pinned reads. Chart s3-notifier-v1.30.1 and later grant the action through the ACK-managed S3ReadAccess policy. The code step's chart bump therefore brings the grant with it. camarades-infrastructure#14 keeps the Terraform-declared policy in step.

Recommended sequence

Step	Change	Approval
0	Merge syrf#4173. Staging gets it automatically; main becomes promotable for the notifier	Normal PR review
1	Production notifier code to the #4173 release (v1.48.0, which staging runs): production config.yaml chartTag, imageTag, lambda.code.sha256, lambda.code.sha256Version. Production stays Suspended. Merge only after the staging import check and the package preservation below	Chris, cluster-gitops PR
2	Versioning Enabled plus the four lifecycle rules, in one values.yaml commit	Chris, cluster-gitops PR, at least a day after step 1 is verified
3	Production Bulk PDF enablement. The production Bulk PDF flag must stay off until versioning (step 2) is verified	Separate decision

The held cluster-gitops PRs need updating before use:

• #1505 has the right step-2 values. Its comment "Only safe while the pre-v1.19 package still runs" is superseded once step 1 is on the #4173 release.
• #1506 and #1507 pin v1.43.0, which predates the fix. v1.43.0 throws on a Suspended bucket, so #1507 would make a rollback of #1505 an outage. Retarget both to the #4173 release, or close them in favour of step 1.

Step 1 draft: production config.yaml

Use s3-notifier-v1.48.0, the release staging has run since 2026-10-15. The first release with #4173 is v1.46.1; its chart is byte-identical to v1.48.0's, and the only notifier change in between is test code and the committed broker permissions. The hash is the one CI recorded for lambda-packages/s3-notifier-v1.48.0.zip, which staging's config.yaml also pins.

 service:
   enabled: true
-  chartTag: s3-notifier-v1.18.0
-  imageTag: "1.17.1" # Placeholder (Lambda uses S3 zip, not Docker image)
+  chartTag: s3-notifier-v1.48.0
+  imageTag: "1.48.0" # Placeholder (Lambda uses S3 zip, not Docker image)
+lambda:
+  code:
+    sha256: FxMEu/vUkkthJ8HGWe/Z57LWnPEn7lVmHDRafLko7fU=
+    sha256Version: 1.48.0

Replace the comment block above chartTag, which explains the v1.18.0 pin, in the same commit.

What the change renders to (2026-10-17: helm template of both versions with production's six ArgoCD value files, in ApplicationSet order, from cluster-gitops origin/main):

• The Bucket CR is byte-identical: versioning stays Suspended, and the lifecycle rules, CORS and notification are unchanged.
• The role's inline S3ReadAccess policy gains s3:GetObjectVersion on syrfapp-uploads/*. Nothing is removed.
• The Function gains packageType: Zip (already on the live CR), a new s3Key and the sha256. Runtime, handler, memory, timeout and the environment the PostSync job writes are unchanged.
• The broker host, user and secret are the same as today (amqp://…:5672, user rabbit). Step 1 does not depend on the AMQPS change (cluster-gitops#1434).
• Messages are wire-compatible with production's API and PM build (104eca9). Search imports and bulk study updates send the same payloads, and the MassTransit SubmitJob and serializer are unchanged between the notifier's 8.5.11 and the services' 8.4.0.

Handler behaviour that changes: - An object with no recognised uploadkind is now logged and ignored. v1.18.0 treated it as a search import and threw. - A malformed recognised record is now logged ("Skipping permanently malformed S3 upload record") and dropped, instead of throwing and being retried. Lambda Errors no longer reveals it; see criterion 3. - Every record in a batch is processed, not only the first. - Search imports now require the screeningimportsettings metadata key (ReferenceUploadRequiredKeys). v1.18.0 did not. The API writes it as an empty string when no screening settings were chosen, which is the common case.

Before merging: 1. Staging import check. Staging runs v1.48.0, but has had no search import since the upgrade. If S3 drops an empty user-metadata value (the notifier's own LocalStack test assumes it may), a search import with no screening settings is silently skipped. Run one staging search import with no screening settings, and confirm it imports and the staging notifier logs show no "Skipping permanently malformed" line. If it fails, do not merge step 1. Fix the writer or the reader first: either the API writes "null" instead of an empty string, as the notifier's LocalStack test already does, or the key comes out of ReferenceUploadRequiredKeys. Then repeat the check. 2. Preserve the running package, so step 1 has a rollback target. No repo key holds it: lambda-packages/production.zip is overwritten on every release, and the v1.17.1 key holds a different package. This reads from and writes to the production AWS account, so it needs Chris's approval, or Chris runs it:

export AWS_REGION=eu-west-1 AWS_PAGER=""
F=syrfAppUploadS3Notifier
KEY=lambda-packages/s3-notifier-production-rollback-pbacxtlOf.zip
# Read-only: download the code the Lambda runs now and compare its hash with the live CodeSha256.
expected=$(aws lambda get-function-configuration --function-name "$F" --query CodeSha256 --output text)
curl -fsS -o rollback.zip "$(aws lambda get-function --function-name "$F" --query Code.Location --output text)"
actual=$(openssl dgst -sha256 -binary rollback.zip | base64)
echo "live=$expected file=$actual"   # both must equal pbacxtlOfMtDi1I9WWNt3tTncMZT/iwdAhANomXSHvM=
# Write-once upload: fails if the key already exists.
aws s3api put-object --bucket camarades-terraform-state-aws --key "$KEY" \
  --body rollback.zip --if-none-match '*'

3. Close cluster-gitops#1506 and #1507 in favour of the step-1 PR.

Step 2 draft: production values.yaml

Exact diff against cluster-gitops main as of 2026-10-14. Rendered with the current chart and the production values, it changes only the Bucket CR: versioning.status goes from Suspended to Enabled, and the three lifecycle additions appear. Function, Role and jobs render byte-identically. The render is identical to that of the held cluster-gitops#1505; only the comments differ.

--- a/syrf/environments/production/s3-notifier/values.yaml
+++ b/syrf/environments/production/s3-notifier/values.yaml
@@ -17,26 +17,24 @@
   # are the *preview/staging* shape and are actively wrong for production.
   # ---------------------------------------------------------------------

-  # Live: aws s3api get-bucket-versioning --bucket syrfapp-uploads
-  #       -> empty output (versioning has NEVER been enabled).
-  # The chart default is `true`, which would silently ENABLE versioning on
-  # the production uploads bucket on first sync. Pinned false to preserve
-  # the live state.
+  # syrf#3669: versioning ENABLED so that Bulk PDF can pin exact object versions.
+  # Was pinned `false` (rendered Suspended) by the ADR-010 Wave 0 cutover to
+  # preserve the never-versioned live state. Objects written before this change
+  # keep version `null`.
   #
-  # Enabling versioning later is a SEPARATE, DELIBERATE change and must not
-  # be done by flipping this flag alone: with versioning on, every object
-  # "expired" by a lifecycle rule below becomes a noncurrent version behind
-  # a delete marker and is retained (and billed) indefinitely. Enabling it
-  # therefore requires adding `noncurrentVersionExpiration` plus an
-  # `expiredObjectDeleteMarker` cleanup rule at the same time — see the
-  # staging values file, which is versioned and carries exactly that shape.
+  # Requires: the notifier on the syrf#4173 release or later (accepts Enabled,
+  # Suspended and never-versioned buckets), and a role with s3:GetObjectVersion
+  # (chart >= v1.30.1). Never run notifier v1.19.0..v1.45.0 on this bucket.
   #
-  # Note: the chart renders `versioning.status` unconditionally, so `false`
-  # produces `Suspended` rather than omitting the field. Suspended and
-  # never-enabled are behaviourally identical (no object versions are ever
-  # retained); the only visible artefact is that get-bucket-versioning will
-  # return {"Status":"Suspended"} instead of empty output.
-  versioning: false
+  # Retention: an overwrite or delete keeps the old bytes as a NONCURRENT
+  # version. expire-noncurrent-versions and remove-expired-delete-markers below
+  # bound that to 30 days (1 day under Projects/_bulk-staging/).
+  #
+  # Rollback: `false` renders Suspended, which is safe with the syrf#4173
+  # release. KEEP EVERY lifecycle rule below when rolling back: they drain the
+  # versions retained while Enabled. Runbook:
+  # syrf docs/how-to/enable-production-upload-bucket-versioning.md
+  versioning: true

   # Live: aws s3api get-bucket-notification-configuration --bucket syrfapp-uploads
   #       -> one Lambda config, events [s3:ObjectCreated:*], filter prefix "Projects/"
@@ -65,18 +63,36 @@
         prefix: ""
       abortIncompleteMultipartUpload:
         daysAfterInitiation: 1
-    # New: reclaim bulk-upload staging ZIPs. The agent deletes each ZIP on
+    # Reclaim bulk-upload staging ZIPs. The agent deletes each ZIP on
     # success, so this only catches objects orphaned by a failure path.
-    # No noncurrentVersionExpiration / expiredObjectDeleteMarker rule is
-    # needed here because this bucket is NOT versioned (see `versioning`
-    # above) — expiry deletes the object outright. If versioning is ever
-    # enabled, this rule MUST gain those fields at the same time.
+    # On a versioned bucket expiry only adds a delete marker, so the
+    # noncurrent ZIP is purged after one more day here.
     - id: expire-bulk-staging
       status: Enabled
       filter:
         prefix: "Projects/_bulk-staging/"
       expiration:
         days: 7
+      noncurrentVersionExpiration:
+        noncurrentDays: 1
+    # syrf#3669: bound what versioning retains. A version becomes noncurrent
+    # when its key is overwritten or deleted; it is removed 30 days later
+    # (the accidental-delete recovery window).
+    - id: expire-noncurrent-versions
+      status: Enabled
+      filter:
+        prefix: ""
+      noncurrentVersionExpiration:
+        noncurrentDays: 30
+    # syrf#3669: remove delete markers with no versions left behind them.
+    # Separate rule: S3 rejects `days` and `expiredObjectDeleteMarker` in one
+    # Expiration block.
+    - id: remove-expired-delete-markers
+      status: Enabled
+      filter:
+        prefix: ""
+      expiration:
+        expiredObjectDeleteMarker: true

   # Live: aws s3api get-bucket-cors --bucket syrfapp-uploads
   #       -> the ten origins below, verbatim and in this order.

Do step 2 at a quiet time, because of the 15-minute first-enablement propagation. Production saw 22 notifier invocations in the week to 2026-09-24. Watch Lambda Errors for 30 minutes afterwards. Any import that failed in that window can be re-uploaded.

Expect one extra lifecycle field. S3 adds a top-level TransitionDefaultMinimumObjectSize: all_storage_classes_128K to the lifecycle configuration, which the values file does not set. The real-AWS rehearsal saw it in both the put-bucket-lifecycle-configuration response and the read-back, with the rules themselves unchanged. ACK may therefore report the Bucket as drifted after the sync. Check that the four rules match (criterion 6), and do not treat the extra field as a rule mismatch.

Rollback

Step 2: versioning

1. In one cluster-gitops commit, set bucket.versioning: false. Keep all four lifecycle rules. With the #4173 release running, Suspended is safe for search imports and bulk study updates. Bulk PDF is refused again, with S3NotifierBulkPdfVersionRequired, so turn the production Bulk PDF flag off before this commit if step 3 has happened. Otherwise new Bulk PDF uploads are refused silently and never complete in the web UI.
2. Verify that get-bucket-versioning returns {"Status":"Suspended"} and that get-bucket-lifecycle-configuration still lists the four rules.
3. The retained versions drain without further action. Noncurrent versions go 30 days after they became noncurrent (1 day under _bulk-staging/), plus lifecycle delay: the rehearsal's 1-day rule took about 2 days. Delete markers go once nothing is left behind them; in the rehearsal that was one daily run after the versions. Billing for each version stops once it becomes eligible.
4. Optional, and only with Chris's approval, because it is irreversible: purge sooner by deleting by version id, in two separate passes: every noncurrent version first, then the delete markers. The order is load-bearing. Deleting a key's current delete marker while versions still sit behind it promotes the newest of them back to current, which un-deletes the object. So a run that stopped part-way, or that interleaved the two lists, would resurrect files users had deleted. Pass 2 starts only after pass 1 has finished, and pass 1 has been re-listed to show no noncurrent versions remain. Never delete a current object version. List each pass first and review the output before deleting:

# Pass 1: noncurrent object versions only. Never current versions, never delete markers.
aws s3api list-object-versions --bucket syrfapp-uploads --output json \
  | jq -c '.Versions[]? | select(.IsLatest | not) | {Key, VersionId}' \
  > pass1-noncurrent-versions.jsonl          # review this file first
while IFS= read -r item; do
  aws s3api delete-object --bucket syrfapp-uploads \
    --key "$(jq -r .Key <<<"$item")" --version-id "$(jq -r .VersionId <<<"$item")"
done < pass1-noncurrent-versions.jsonl

# Gate: must print 0 before pass 2 starts.
aws s3api list-object-versions --bucket syrfapp-uploads --output json \
  | jq '[.Versions[]? | select(.IsLatest | not)] | length'

# Pass 2: delete markers. Each now has nothing behind it, so removing it restores nothing.
aws s3api list-object-versions --bucket syrfapp-uploads --output json \
  | jq -c '.DeleteMarkers[]? | {Key, VersionId}' \
  > pass2-delete-markers.jsonl               # review this file first
while IFS= read -r item; do
  aws s3api delete-object --bucket syrfapp-uploads \
    --key "$(jq -r .Key <<<"$item")" --version-id "$(jq -r .VersionId <<<"$item")"
done < pass2-delete-markers.jsonl

If uploads continue during the purge, a new overwrite or delete between the gate and pass 2 can create a fresh noncurrent version behind a marker. Re-run the gate immediately before pass 2, and if it is not 0, repeat pass 1 first.

1. Keep the lifecycle rules afterwards. On a Suspended bucket they cost nothing, and they clean up if versioning is ever re-enabled.

Objects written while Enabled keep their real version ids as current versions until they are next overwritten or deleted. That is harmless: they are single current objects, and the fixed notifier reads new events by key.

Step 1: notifier code

Do not revert config.yaml to chart v1.18.0: that does not roll the code back. The v1.18.0 chart has no lambda.code.sha256 support, and the ACK controller pushes code only when the spec hash differs from the live CodeSha256. Reverting would leave the v1.48.0 code running and only remove s3:GetObjectVersion.

Stay on chart s3-notifier-v1.48.0 and point it at the package preserved before step 1. With an explicit s3Key the chart skips its version check. This shape was rendered on 2026-10-17:

 lambda:
   code:
-    sha256: FxMEu/vUkkthJ8HGWe/Z57LWnPEn7lVmHDRafLko7fU=
-    sha256Version: 1.48.0
+    s3Key: lambda-packages/s3-notifier-production-rollback-pbacxtlOf.zip
+    sha256: pbacxtlOfMtDi1I9WWNt3tTncMZT/iwdAhANomXSHvM=

sha256Version is dropped on purpose: with an explicit s3Key the chart skips its version binding, so the field would only record provenance. The extra s3:GetObjectVersion grant stays, which is harmless. This is valid in both bucket states. Never roll back to a build from v1.19.0 up to the last release before #4173.

Rehearsal

LocalStack (done, 2026-10-14)

rehearse-bucket-versioning-localstack.sh runs against a disposable LocalStack container and never touches AWS. It routes ObjectCreated:* events for Projects/ to SQS and matches each event to its write by eTag. Result on LocalStack 4.13.1:

Step	PutObject VersionId	Event versionId	Versions after the step
1. Never versioned	none	absent	null
2. Suspended without ever being Enabled (production today)	none	absent	null
3. Enabled: write, then overwrite	two distinct real ids	the matching real id each time	new current, previous real id noncurrent, old null noncurrent
4. Proposed lifecycle rules			all four rules accepted and returned unchanged; Days with ExpiredObjectDeleteMarker in one rule is rejected
Delete while Enabled			delete marker added
5. Rollback to Suspended, then write	none	absent	new null current; both real-id versions and the delete marker remain
6. Optional purge: pass 1 noncurrent versions, gate, pass 2 delete markers			gate reads 0; only the current null version remains

What the rehearsal does not show:

• Lifecycle expiry. LocalStack stores lifecycle rules but does not run them. The real-AWS timings are in the results table below.
• The literal "null": now answered. LocalStack omits versionId for Suspended writes. The real-AWS rehearsal showed that real S3 does the same in every state that is not Enabled, so the notifier should never see "null". It still handles both, as defence in depth. Its unit tests use the events/bucket-modes/ fixtures, and its LocalStack integration tests send events both without versionId and with "null".

Real AWS scratch bucket

Chris approved this on 2026-10-14. It ran in eu-west-1 against a new bucket and queue, syrf-versioning-rehearsal-20261014. Both were deleted on 2026-10-17 at 18:17Z, once both expiry times had been recorded (head-bucket returns 404; get-queue-url returns NonExistentQueue). Never use syrfapp-uploads or any SyRF bucket for this.

1. Create the bucket and an SQS queue whose policy lets that bucket send to it. Add an s3:ObjectCreated:* notification with prefix Projects/ to the queue.
2. Put one object and read the event. Suspend, put again, and read the event.
3. Enable, then wait 15 minutes. Put, overwrite and delete one key. Expect two real ids in the events, and two noncurrent versions plus one delete marker in list-object-versions: the overwrite makes write 1 noncurrent, and the delete makes write 2 noncurrent.
4. Apply the four rules with noncurrentDays: 1 in place of 30. Suspend. Put a new key, which is the rollback state, and read its event.
5. Check list-object-versions at least daily. Record when the noncurrent versions go, and then when the expired delete marker goes.
6. Delete every version, the bucket and the queue.

Results:

Step	Bucket state	Observed
2	Never versioned	PutObject returns no VersionId; the event has no versionId key
2	Suspended, never Enabled (production today)	The same: no VersionId, and no versionId key in the event. It is not "null"
3	Enabled	Each put returns a real id, and its event carries the same id. The delete adds a delete marker. list-object-versions shows two noncurrent versions plus one delete marker
4	Lifecycle	The four rules are accepted and read back unchanged, apart from the added TransitionDefaultMinimumObjectSize. Days with ExpiredObjectDeleteMarker in one Expiration is rejected with MalformedXML
4	Suspended after being Enabled (rollback)	The new key's PutObject returns no VersionId, its event has no versionId key, and it is stored as version null
5	Noncurrent expiry	The versions became noncurrent at 14:46:53Z and 14:47:16Z on 2026-10-14, so were eligible from 2026-10-16T00:00Z. They were removed between 12:51:51Z and 18:17:08Z on 2026-10-16
5	Expired delete marker	Removed between 12:17:07Z and 18:17:05Z on 2026-10-17: one daily lifecycle run after the noncurrent versions went, about 3 days after the delete
5	Current versions	Both current null objects stayed untouched throughout

Acceptance criteria

#	Criterion	Verification
1	Main-line notifier code processes search imports and bulk study updates from a never-versioned and a Suspended bucket	S3EventObjectVersionTests and UnversionedBucketLocalStackIntegrationTests pass (#4173)
2	Bulk PDF is never dispatched or captured without a real version	Evaluate_PerBucketModeAndUploadKind, Evaluate_UnversionedReservedPrefixObject_IsRefused and UnversionedBucket_BulkPdfUpload_IsRefusedWithoutDispatch pass
3	Step 1 runs the #4173 release in production, on a Suspended bucket, without errors	aws lambda get-function-configuration --function-name syrfAppUploadS3Notifier --query CodeSha256 equals the release hash; the next production upload logs S3NotifierUnversionedFallback and is imported; Lambda Errors stays 0 for 7 days; and a CloudWatch Logs search over the same 7 days finds neither "Skipping permanently malformed S3 upload record" nor "Unrecognised or missing 'uploadkind'" (v1.48.0 logs and drops these instead of erroring)
4	The cost baseline is recorded before step 2	BucketSizeBytes and NumberOfObjects for the previous 14 days, and the list-object-versions summary, are pasted into the step-2 PR
5	The production role can read versions before or with step 2	aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::318789018510:role/syrfS3NotifierProductionLambdaRole --action-names s3:GetObjectVersion --resource-arns arn:aws:s3:::syrfapp-uploads/Projects/x returns allowed
6	Step 2 enables versioning together with all four lifecycle rules	get-bucket-versioning returns Enabled; get-bucket-lifecycle-configuration returns exactly Incomplete Multipart Cleanup, expire-bulk-staging (with noncurrent 1), expire-noncurrent-versions (30) and remove-expired-delete-markers; the notification (ObjectCreated:*, Projects/), CORS (10 origins) and public/* policy are unchanged
7	After step 2, production events are version-pinned	The next production upload logs no S3NotifierUnversionedFallback; Lambda Errors stays 0
8	Retained versions stay bounded	At D+35, BucketSizeBytes is within 2 GB of the pre-enable trend, and list-object-versions shows no noncurrent version older than 32 days
9	The rollback is rehearsed	The LocalStack rehearsal output matches the table above (done); the real-AWS rehearsal is recorded with observed expiry times before step 2 (done: run from 2026-10-14 to 2026-10-17; noncurrent versions and the expired delete marker both expired as configured, and both times are in the results table)
10	Rollback keeps the cleanup rules	The rollback commit changes only versioning; get-bucket-lifecycle-configuration still returns the four rules afterwards
11	The production Bulk PDF flag stays off until versioning is verified	Production web/API values have no featureFlags.bulkPdfUpload: true until criteria 6 and 7 hold; any rollback of step 2 turns it off first

Follow-ups (not in this change)

• A CloudWatch Logs metric filter and alarm on S3NotifierUnversionedFallback and S3NotifierBulkPdfVersionRequired, and an alarm on BucketSizeBytes. Each needs a declarative owner first.
• syrf#3923: staging's versioned bucket has no noncurrent expiry.
• Retarget or close cluster-gitops#1506 and #1507 (v1.43.0 predates the fix).