Service Promotion Workflow

Guide to promoting services from staging to production using GitOps best practices.

Overview

The promotion workflow is the process of moving a validated service version from staging to production. This follows a progressive delivery model:

Development → Staging → Production
    ↓            ↓          ↓
  Code Merge   Auto Deploy  Manual Promotion

Promotion Philosophy

Key Principles: 1. Staging is automatic - Deployments to staging happen automatically after CI/CD completes 2. Production is manual - Promotions to production require explicit approval and testing 3. Version matching - Production should only run versions that have been validated in staging 4. Rollback ready - Always have a plan to rollback if issues arise 5. Communication - Notify team of production deployments

Prerequisites

• Service successfully deployed and tested in staging
• Access to cluster-gitops repository
• Production deployment checklist completed
• Stakeholder approval (for major changes)

Promotion Process

Step 1: Validate in Staging

Before promoting, ensure the service is stable in staging:

# 1. Check pod status
kubectl get pods -n syrf-staging -l app=api

# 2. Check recent logs for errors
kubectl logs -n syrf-staging -l app=api --tail=200 | grep -i error

# 3. Verify health endpoints (if configured)
curl https://staging-api.syrf.org.uk/health || echo "No health endpoint configured"

# 4. Check current version
yq eval '.service.chartTag' syrf/environments/staging/api/config.yaml

# 5. Check ArgoCD sync status
argocd app get api-staging

Staging Validation Checklist: - [ ] Pods are running and healthy - [ ] No error logs or warnings - [ ] Health endpoint returns 200 OK - [ ] Smoke tests pass (if available) - [ ] No performance degradation - [ ] Feature works as expected - [ ] Database migrations successful (if any)

Step 2: Promote to Production

Updated 2025-11-16: Production promotion is fully automated with manual review gate

Production promotion happens automatically after successful staging deployment:

1. Trigger: After staging promotion succeeds in CI/CD workflow
2. Job: promote-to-production job starts automatically
3. Action: Copies all service configs from staging to production
4. Updates: Sets envName: production and keeps chartTag from staging
5. Creates PR: Labeled requires-review with review checklist
6. Manual Gate: Administrator must review and merge PR manually
7. CI/CD Completes: Workflow shows green checkmark after PR creation
8. Deployment: After PR merge, ArgoCD auto-syncs to production

Key Points: - No manual workflow trigger needed - No GitHub Environment configuration required - Manual gate happens at PR merge step in cluster-gitops repo - All services promoted together with matching versions - See PRs: https://github.com/camaradesuk/cluster-gitops/pulls?q=is:pr+label:requires-review

Example config file updated:

# syrf/environments/production/api/config.yaml
serviceName: api
envName: production  # ← Changed from staging
service:
  enabled: true
  chartTag: api-v9.1.1  # ← Same version as staging

Manual Review Process

When the automated PR is created:

1. Review the PR at cluster-gitops repository
2. Check the staging validation checklist
3. Verify the version matches staging
4. Merge the PR to trigger production deployment

# View the production promotion PR
cd cluster-gitops
gh pr list --label requires-review

# Review changes in the PR
gh pr diff <PR_NUMBER>

# Check what's being promoted
git diff main...production-promotion-<run-id> -- syrf/environments/production/

# Merge after review
gh pr merge <PR_NUMBER> --merge

A promotion that changes a config.yaml also runs the Broker permissions parity workflow (.github/workflows/broker-permissions-parity.yml). It compares the RabbitMQ grants in plugins/helm/rabbitmq/resources/rabbitmq-definitions.yaml with each promoted service's broker-permissions.json at the promoted commit (scripts/broker-permissions-parity/README.md). It is not a required check. Read its job summary: a principal that passed on main and fails on the promotion means the promoted commit changed that service's broker contract, and the definitions must follow before the new code relies on the change. Planned, tracked gaps (known_gap in plugins/helm/rabbitmq/broker-permissions-parity.yaml) are reported as KNOWN and do not fail. If a promotion closes one, the check fails with "known gap closed — remove the known_gap entry": remove that entry, in this PR or straight after it.

Step 3: Monitor Production Deployment

Critical: Closely monitor the production deployment

# 1. Watch pod rollout
kubectl get pods -n production -l app=api -w

# 2. Check deployment progress
kubectl rollout status deployment/api -n production

# 3. Monitor logs in real-time
kubectl logs -n production -l app=api --tail=100 -f

# 4. Verify health endpoint
curl https://api.syrf.org.uk/health

# 5. Check for errors in last 5 minutes
kubectl logs -n production -l app=api --since=5m | grep -i error

Production Monitoring Checklist: - [ ] New pods starting successfully - [ ] Old pods terminating gracefully - [ ] No error logs during rollout - [ ] Health endpoint responding - [ ] User traffic flowing normally - [ ] No alerts firing (if monitoring set up)

Step 4: Post-Deployment Verification

After deployment completes, perform verification:

# 1. Verify correct image version deployed
kubectl get deployment api -n production -o jsonpath='{.spec.template.spec.containers[0].image}'

# 2. Check all replicas are ready
kubectl get deployment api -n production

# 3. Test critical user flows
# Example: Login, create project, run search, etc.

# 4. Check application metrics (if available)
# Navigate to: https://grafana.syrf.org.uk/d/production-api

# 5. Verify no increase in error rates
# Check logs, metrics, and user reports

Rollback Procedure

If issues are detected post-deployment, rollback immediately:

Quick Rollback (Recommended)

# 1. Revert to previous version
cd cluster-gitops
nano environments/production/services/api.yaml

# 2. Change chartTag to previous version
service:
  chartTag: api-v1.2.2  # Previous stable version

# 3. Commit and push
git commit -am "rollback(api): revert production to v1.2.2 due to [issue]"
git push

# 4. Monitor rollback
kubectl rollout status deployment/api -n production
kubectl logs -n production -l app=api --tail=100 -f

Git-Based Rollback

# Alternative: Use git revert
git log --oneline -- environments/production/services/api.yaml
git revert <commit-sha>
git push

Kubernetes Native Rollback

# Last resort: kubectl rollback (doesn't update git state!)
kubectl rollout undo deployment/api -n production

# IMPORTANT: Update git to match cluster state
nano environments/production/services/api.yaml
# Update chartTag to match what's running
git commit -am "chore(api): sync git with rolled-back production state"
git push

Service-Specific Promotion Examples

API Service

# Staging validation
kubectl logs -n staging -l app=api --tail=200
curl https://staging-api.syrf.org.uk/health

# Promote (automated workflow recommended)
# OR manually:
nano environments/production/services/api.yaml
# Update: chartTag: api-v8.21.0
git commit -m "promote(api): v8.21.0 → production"
git push

# Monitor
kubectl rollout status deployment/api -n production

Web Service

# Staging validation
kubectl logs -n staging -l app=web --tail=200
curl https://staging.syrf.org.uk

# Promote (automated workflow recommended)
# OR manually:
nano environments/production/services/web.yaml
# Update: chartTag: web-v5.0.1
git commit -m "promote(web): v5.0.1 → production"
git push

# Monitor
kubectl rollout status deployment/web -n production

Project Management Service

# Staging validation
kubectl logs -n staging -l app=project-management --tail=200
curl https://staging-api.syrf.org.uk/pm/health

# Promote (automated workflow recommended)
# OR manually:
nano environments/production/services/project-management.yaml
# Update: chartTag: pm-v10.45.0
git commit -m "promote(pm): v10.45.0 → production"
git push

# Monitor
kubectl rollout status deployment/project-management -n production

Best Practices

Timing

• Avoid peak hours: Deploy during low-traffic periods (e.g., evenings, weekends)
• Business hours preferred: Deploy when team is available to respond to issues
• Coordinate with team: Notify team members before production deployments
• Maintenance windows: Use scheduled maintenance windows for risky changes

Communication

Before Deployment:

🚀 Production Deployment Planned

Service: API v1.2.3
Time: 2025-11-12 18:00 UTC
Changes: User authentication improvements
Validated in staging: ✅ All tests passing
Risk level: Low
Rollback plan: Revert to v1.2.2

After Deployment:

✅ Production Deployment Complete

Service: API v1.2.3
Deployed at: 2025-11-12 18:15 UTC
Status: Healthy - no errors detected
Monitoring: https://grafana.syrf.org.uk/d/api

Testing

• Always validate in staging first (no shortcuts!)
• Run smoke tests before promoting
• Test critical user flows post-deployment
• Monitor for at least 30 minutes after deployment
• Have rollback command ready before deploying

Version Management

• Never skip versions - Deploy 1.2.3 → 1.2.4, not 1.2.3 → 1.3.0 (unless tested)
• Tag format consistency - Always use semantic versions (no latest in production)
• Git history - Maintain clear commit messages with rationale
• Documentation - Update CHANGELOG or release notes

Troubleshooting

Promotion Blocked by ArgoCD

Symptom: Production application shows "OutOfSync" but won't sync

# Check sync status
argocd app get api-production

# Check for sync policy issues
kubectl get application api-production -n argocd -o yaml

# Force sync if needed
argocd app sync api-production --force

Rollout Stuck

Symptom: New pods not starting or old pods not terminating

# Check rollout status
kubectl rollout status deployment/api -n production

# Check pod events
kubectl describe pod <pod-name> -n production

# Check for resource constraints
kubectl top pods -n production
kubectl describe nodes

# Manual intervention (use cautiously)
kubectl rollout restart deployment/api -n production

Health Check Failures

Symptom: Readiness probe failing, traffic not routing to new pods

# Check probe configuration
kubectl describe deployment api -n production | grep -A 5 Readiness

# Test health endpoint from within cluster
kubectl exec -n production deployment/api -- curl -f http://localhost:8080/health

# Check logs for startup errors
kubectl logs -n production <new-pod-name> --tail=200

Related Documentation

• Deploying Services - How to deploy to staging
• Environment Configuration - Managing environment values (TODO)
• Troubleshooting - Common deployment issues (TODO)
• CI/CD Workflow - How images are built
• Cluster Configuration - Production cluster details

External Resources

• Progressive Delivery: https://www.weave.works/blog/what-is-progressive-delivery-all-about
• Argo Rollouts: https://argoproj.github.io/argo-rollouts/
• Blue/Green Deployments: https://martinfowler.com/bliki/BlueGreenDeployment.html