Skip to content

Juniper runner routing

This is the per-job routing ledger for the shared self-hosted juniper-ci runner pool: which GitHub Actions jobs run self-hosted, which stay GitHub-hosted, and the controls that make each decision safe. It was relocated verbatim from the root CLAUDE.md.

Read this before changing any runs-on: value, and run .github/scripts/validate-workflows.sh afterwards — every route below is pinned by a focused contract script that fails CI when the pin drifts. The quick-reference summary lives in .claude/rules/ci-workflows.md.

PR application checks follow product paths and concrete execution/routing contracts. Editing validate-workflows.sh alone (for example, updating a Claude review contract) runs workflow validation without selecting .NET, Angular, integration or application scan jobs. Changes to pr-tests.yml, shared runner cleanup, or a dedicated PR routing contract still select their live lanes; combining validator edits with product changes preserves the product checks. .github/scripts/test-pr-change-detection.sh executes the actual detector with fixtures for both cases. This changes job selection only: required checks and runner security controls are unchanged.

Shared Juniper runner boundary

Default posture: prefer the shared juniper-ci self-hosted pool where possible; GitHub-hosted (ubuntu-latest) is the exception that needs a reason. Hosted minutes are metered and the org Actions spending limit was exhausted twice on 2026-08-17/18, each time blocking every hosted job across every repo (the signature is a 2-3s failure with steps: 0 and no logs — gh api repos/camaradesuk/syrf/check-runs/<job-id>/annotations names the cause). Self-hosted consumes no budget and is generally faster.

The reasons that still justify staying hosted, and the only ones that do by default:

  1. Fork-originated code. Untrusted PR code must never run on the shared host. The *-fork hosted equivalents, the same-repository head gates, contents: read, and non-persisted checkout credentials all stay exactly as they are.
  2. Undeclared host-tool dependencies. Check before routing (see the helm rollback below); install the tool via a camaradesuk/server-config role or stay hosted, or, for a job-scoped CLI binary, a pinned checksum-verified runtime install as the Preview jobs do (see the 2026-08-31 record).
  3. A specific, written reason recorded here — not inherited caution.

Holding a secret is not by itself a permanent disqualifier — the pool already runs the Claude agent jobs with an OAuth token and write authority — but it is only acceptable on top of three controls together, not trigger-gating alone:

  1. Host-level listener isolation — a prerequisite of the host, not something CI can check. Secret-bearing jobs became acceptable on Juniper only once every listener enforced PrivateTmp, MemorySwapMax=0, per-listener CPU/memory/task caps and post-job cleanup; before that the listeners had no isolation whatsoever. The active persistent pool has twelve Pomegranate listeners. All carry juniper-normal-ci, juniper-elastic, and pomegranate-ci; only pomegranate-01 also carries the exclusive pomegranate-e2e label. Ordinary jobs must therefore request juniper-normal-ci as well as the group, while each specialized route must request its complete workload-specific label set. Their provisioning and runtime/tool baseline are owned by camaradesuk/server-config, and they must be online and revalidated before a SyRF route is enabled. That history is recorded as rationale in validate-workflows.sh (~line 895), but the script cannot assert host controls — it only checks workflow-YAML-observable properties. So this control is verified by the camaradesuk/server-config roles that provision the listeners, and a credentialed job must not run outside the audited group — which is the shared CAMARADES group juniper-ci, and only that group. Routing a secret-bearing job to any other self-hosted group satisfies the letter of this rule while violating it.
  2. Trusted triggers — the unchanged per-event author_association gates for claude.yml, and the /claude-review PR-comment gate for claude-code-review.yml. The latter executes default-branch workflow code and checks maintain/admin access for requester and rerunner plus open, non-draft, same-repository/default-base PR eligibility before checkout or OAuth use. No labels or PR lifecycle events activate reviews.
  3. Least-privilege permissions: and checkout authentication appropriate to each lane. Ordinary test lanes do not persist credentials. The two Claude lanes retain checkout authentication because the action fetches before configuring its own App token in progress mode (upstream #1340); the review lane uses contents: read, and checkout removes the credential post-job. No custom GitHub token overrides the action’s App OIDC exchange.

The per-job contracts below remain binding and take precedence over this preamble: where a specific bullet names a job as hosted, that bullet governs until that job is individually reviewed and its bullet updated. This preamble changes the default for new or unreviewed work; it does not silently re-route anything already pinned below.

2026-10-15 Bramble native routes (superseding routing record)

Approved by Chris on 2026-10-15 to take Docker-free work off the shared pool: Juniper was at a load of about 33 on 48 threads, with runner losses and timeouts, while Bramble sat at about 4.6 on 20. This record supersedes the "on Juniper" wording in the bullets below for the jobs it names.

  • Listeners. bramble-01 and bramble-02 on Chris's workstation, in the shared juniper-ci group, provisioned by camaradesuk/server-config roles/bramble (runbook). They register with --no-default-labels and advertise only the existing org label bramble-native-ci, so the route is exactly runs-on: {group: juniper-ci, labels: [bramble-native-ci]}. Requesting self-hosted, linux or x64 as well would match no listener. No new group or label was added.
  • Host boundary. Each listener is its own systemd service and account with CPUQuota=300%, MemoryMax=16G (MemoryHigh=12G), MemorySwapMax=0, TasksMax=1024, ProtectSystem=strict, ProtectHome, NoNewPrivileges, an empty capability set, private devices, /tmp bound into its own 64 GiB filesystem, and InaccessiblePaths=/run/docker.sock /run/containerd /data. There is no sudo and no Docker. Host tools verified on 2026-10-15 (read-only ssh bramble): Node 22.22, .NET SDK 10.0.112 only (no .NET 8), Helm 3.21.3, git, gh, jq, shellcheck, python3 3.14 with PyYAML; no yq, pnpm, pip or browser. Jobs install Node, pnpm and Helm through their pinned actions into the listener's own tool cache and home, exactly as on Juniper.
  • Routes (all in pr-tests.yml except the last): validate-generated-code, validate-pdf-path-case-folding, validate-openapi-spec, test-charts, check-gsd-state, and test-statistics-scripts.yml:unit. All are Docker-free and read-only, keep their same-repository gate (unit also runs on push to main), their hosted *-fork twin, exactly contents: read, no secrets, persist-credentials: false with checkout first, their exact timeouts, their run-unique runner.temp state and their always() delegation to juniper-job-cleanup.sh (unchanged; it derives everything from GITHUB_WORKSPACE and RUNNER_TEMP). validate-openapi-spec also exports DOTNET_INSTALL_DIR beneath its job root before setup-dotnet, because the action's default /usr/share/dotnet is read-only there and NSwag needs the .NET 8 runtime the host lacks; the toolchain cleanup removes that install with the rest of the job root.
  • Measured durations (PR runs 2026-10-07 to 2026-10-15 on the shared pool, successful jobs): OpenAPI median 4.8 min (p90 6.4, 125 runs in 7.7 days), generated code 0.8, Unicode artifact 0.9, charts 0.6, GSD 0.2, statistics unit 0.3. Replaying those arrivals against two Bramble slots gave a median queue wait of zero, p90 about 6 minutes and p99 about 18 minutes, with 7 of 117 runs finishing after their own Test .NET job (most by under two minutes). Even assuming Bramble is 1.3x slower plus a 90-second .NET download, the median run waits nothing; bursts of many pushes at once (merging main into every worktree) are where queueing shows up.
  • Pinned by EXPECTED_BRAMBLE_NATIVE_ROUTES in validate-workflows.sh (exact route, no Docker step, service container or job container, fork gate, read-only permissions, checkout first, always() cleanup; any other bramble* label fails), test-pr-validation-gates-runner-routing.sh (labels, OpenAPI install directory and its ordering before setup-dotnet) and test-self-hosted-migration-routing.sh. Changing pr-tests.yml or the validation-gate contract selects every moved gate; changing test-statistics-scripts.yml triggers unit.
  • Deliberately not moved. Anything using Docker or Testcontainers (test-dotnet, test-dotnet-mongo-data, Identity/Quartz integration, main integration), every E2E, Sonar, preview, deploy, release, promotion or production job, and every fork lane. test-web stays on the shared pool for now: its nginx proxy step needs Docker (it would have to become its own Juniper job), it would hold one of only two Bramble slots for 12-20 minutes, and its memory peak on a 16 GiB listener is unmeasured. The ci-cd.yml main release tests stay too, so a workstation outage cannot hold a release. validate-workflows.yml and docs-validation.yml are GitHub-hosted, so they add no Juniper load; moving them would need hosted fork twins and tools Bramble lacks (yq, actions/setup-python 3.11).
  • Availability. Bramble is a workstation. If both listeners are offline, the moved gates queue and the required Test Summary waits with them. Main releases are unaffected.
  • Host backstop gap. Bramble listeners have no root-owned ACTIONS_RUNNER_HOOK_JOB_COMPLETED purge like the Juniper and Pomegranate listeners, so the in-job always() cleanup is the only residue removal, and the listener's tool cache and home (setup-node, setup-pnpm, Helm) persist between jobs inside the 64 GiB filesystem. Adding the completion hook is a camaradesuk/server-config change.

2026-08-31 paid-minute migration (superseding routing record)

This record supersedes any older “remain hosted” statement in the dated bullets below for the jobs it names. General same-host or abstract trust-boundary concerns do not block internal same-repository work; the concrete boundary is that fork-authored executable code must not reach the persistent pool with internal credentials.

  • Preview version calculation (_preview-gitversion.yml:version) and reusable image building (_docker-build.yml:build) use juniper-ci with [self-hosted, linux, x64, juniper-normal-ci]. The Angular artifact build, Lambda packaging, image retagging, Git tags, deployment/status/GitOps orchestration, Argo verification, failure reporting, label repair, and authorized reseed jobs in pr-preview.yml use the same route. The remaining hosted Preview jobs are exactly check-label-hosted, check-label, and cleanup-tags: deterministic fork denial, its fork-capable selector, and the not-yet-proven multi-cloud teardown lane. Fork preview builds are denied without checkout; create-deployment is same-repository-only; /reseed-db verifies OWNER/MEMBER/COLLABORATOR association and same-repository provenance before any credential-bearing step.
  • The preview-orphan-sweep runtime-image workflow (preview-orphan-sweep-image.yml:build and :publish) uses the same juniper-ci / [self-hosted, linux, x64, juniper-normal-ci] route as _docker-build.yml:build: it is a same-repository Docker build with docker/build-push-action, gated to push on main / workflow_dispatch for the GHCR publish step, and needs only the host Docker daemon the image builds already rely on. The build job is same-repository-only; fork pull requests run the identical steps on the hosted build-fork lane, and test-preview-orphan-sweep-image-contract.sh pins both gates and the exact 20-minute timeouts (2026-09-15, syrf#2627).
  • The Preview GitOps jobs refresh-metadata, write-versions, reseed-database, and wait-for-health provision yq (pinned v4) and kubectl (pinned to the cluster's control-plane minor, ±1 skew accepted) at runtime: an Install yq / Install kubectl step downloads a pinned release binary into $RUNNER_TEMP, verifies its published SHA-256, and adds it to GITHUB_PATH. This is the documented exception to "install via server-config or stay hosted" for a job-scoped CLI tool that needs no daemon, no root, and no host state — mikefarah/yq@v4 is a Docker action and never puts yq on PATH; GitHub's hosted image preinstalls both yq and kubectl, Juniper does not. Bumping either pin means updating the version and re-checking the checksum source; tools that need root, a daemon, or host state (Docker, Helm, the gcloud SDK) still go through server-config.
  • Same-repository PR validation now also includes validate-pdf-path-case-folding and test-identity-integration on ordinary Juniper capacity. Their explicit *-fork counterparts remain on ubuntu-latest. test-quartz-integration uses one hosted internal/fork lane: its SQL Server container exited during startup in two clean Pomegranate capability runs on PR #3027, so moving it would weaken required coverage. The shared detect-changes and tokenless test-summary jobs remain hosted because one output/fan-in graph serves both internal and fork routes.
  • sonar-web joins sonar-api, sonar-pm, sonar-quartz, and sonar-identity on the Pomegranate subset, requesting [self-hosted, linux, x64, juniper-normal-ci, pomegranate-ci]. All five are same-repository-only, token-minimal, full-history, non-persisting checkouts with isolated state and always() cleanup. Forks cannot launch Sonar.
  • The E2E heavy body runs on the two dedicated E2E listeners by host-specific label (syrf-e2e-pomegranate-01, syrf-e2e-bramble-vm-01); its base-defined, no-checkout check-label, comment, and cleanup jobs use ordinary Juniper capacity. pull_request_target keeps fork code out of every self-hosted E2E job.

  • .github/scripts/test-self-hosted-migration-routing.sh, the lane contracts, and EXPECTED_GLOBAL_SELF_HOSTED_ROUTES are the executable inventory. CodeQL routing is intentionally unchanged.

  • Both pr-tests.yml .NET unit routes retain the full Release solution build and analyzers but pass GenerateOpenApi=false. The matching dedicated OpenAPI routes own the API build, NSwag execution, and generated-contract diff, so do not restore duplicate build-time generation to the unit lanes.

  • Main release unit tests are split the same way as the PR lane (#3886, mirroring #4127): test-dotnet ("Test .NET Services") runs every test project except SyRF.ProjectManagement.Mongo.Data.Tests, from a throwaway copy of syrf.sln with that project removed, with a 35-minute budget, and test-dotnet-mongo-data ("Test .NET Services (Mongo.Data)") runs that project alone with a 50-minute budget. Both share one trigger (including the run_trusted_tests_only canary), route, contents: read permission and four-processor cap; test-dotnet verifies Helm and keeps the full-solution Release build, test-dotnet-mongo-data verifies Docker and builds only its project. release-gate needs both and blocks on a failure or cancellation in either. .github/scripts/test-main-dotnet-runner-routing.sh pins all of this exactly, with one-minute fixtures either side of each budget. The single job had a 45-minute budget (raised from 25 on 2026-10-02): once Mongo.Data grew to about 27 minutes it took 33–36 minutes on most listeners and was cancelled at the limit on the slower Pomegranate ones with no failing test (release run 38694255725, both attempts), which held staging. Before that, a healthy run took about 22–23 minutes, so 25 left no headroom: release run 37007594525 was cancelled at 25:35 and blocked a staging deploy. Release run 35039263554 exceeded the former 15-minute limit twice after restore/build consumed most of the budget; the test selection and release gates remain unchanged. Main integration tests have the same 45-minute budget (raised from 25 on 2026-10-14): Juniper listeners finish in about 22–24 minutes, but from 2026-10-03 to 2026-10-14 the 25-minute limit cancelled 32 of 33 runs that landed on Pomegranate listeners, which build and test more slowly, and each cancellation held the production PR.
  • The two self-hosted Mongo.Data test steps (ci-cd.yml test-dotnet-mongo-data / Run unit tests, pr-tests.yml test-dotnet-mongo-data / Run tests) run dotnet test under .github/scripts/cgroup-memory-sampler.sh run -- (#4246). The runner's systemd unit is limited to 12 GiB (MemoryMax, no swap), and a normal run peaks near 2 GiB, so an OOM kill of the unit (it cancels the job and no artifact uploads) means something unusual filled the cgroup. The sampler prints [mem-sampler] lines to the step log: the limited cgroup and its memory.high/memory.max/memory.swap.max once, then every 30 s (MEM_SAMPLER_INTERVAL_SECONDS) memory.current, memory.peak, anon/file/shmem, the oom/oom_kill counts, host MemAvailable and the top 8 processes by RSS in that cgroup and its descendants. After an OOM, read the last [mem-sampler] ---- block in the cancelled step's log: a memory.current near 12 GiB with a huge testhost/dotnet row means a test allocated without bound; a large row with a long elapsed and an unrelated command line is a leftover process from an earlier job on that listener; large shmem or file with small RSS rows means tmpfs or page cache, not a process. It is fail-safe (cgroup v1, a missing file or no permission prints one unavailable line and the tests still run) and returns the test command's exit status unchanged. The hosted fork lane has no such limit and runs without it; test-pr-dotnet-runner-routing.sh and test-main-dotnet-runner-routing.sh pin both facts, and test-cgroup-memory-sampler.sh (Validate Workflows) tests the script.
  • The self-hosted routes in ci-cd.yml are test-dotnet, test-dotnet-mongo-data, test-dotnet-integration, and test-web. All four must use the shared CAMARADES group juniper-ci with labels [self-hosted, linux, x64, juniper-normal-ci], run only on refs/heads/main, keep permissions at exactly contents: read, avoid all secrets-context references, and check out with persisted credentials disabled. The workflow root and its always() summary exception must fail closed outside the main ref so the whole graph stays main-only. The no-checkout release-gate and production-gate orchestration jobs (added 2026-10-03) use the same group and labels with no checkout, no action steps, permissions: {}, no secrets, an exact 5-minute timeout, a top-level main-ref conjunct and the trusted-test canary deny; validate-workflows.sh pins them as separate orchestration routes and lists ci-cd.yml:release-gate and ci-cd.yml:production-gate in EXPECTED_GLOBAL_SELF_HOSTED_ROUTES. With no workspace writes there is nothing for juniper-job-cleanup.sh to remove.
  • The automatic PR test routes on self-hosted capacity are the read-only pr-tests.yml validation gates (validate-generated-code, validate-openapi-spec, test-charts, and check-gsd-state, on the Bramble native listeners since 2026-10-15; see that record above) plus test-web, test-dotnet and test-dotnet-mongo-data on Juniper; they are an explicitly accepted shared-host risk and are pinned to same-repository PR heads. The lightweight detect-changes and no-checkout test-summary orchestration jobs remain hosted so one output graph can select the Juniper jobs for same-repository heads and explicit GitHub-hosted *-fork counterparts for fork heads. Fork source must never be checked out or executed on a persistent listener. Every checked-out Juniper route stays at exactly contents: read, has no secret/OIDC/write authority, uses non-persisted checkout credentials, has a bounded timeout, and delegates its always() cleanup through .github/scripts/juniper-job-cleanup.sh. The seven exact invocations (the original six lanes plus test-dotnet-mongo-data) are pinned in the validation-gate, Web and .NET routing contracts; hosted-fork jobs do not invoke it. The helper replaces the Actions step shell with exec, arms trap 'cancelled=1' TERM INT before meaningful work, derives deletion roots internally from RUNNER_TEMP plus lane-owned basename-only keys, captures each command status, and aggregates final absence and cleanliness proofs into one verdict. Its fixed standard, toolchain, and workspace profiles prevent workflow callers from supplying cleanup commands or arbitrary paths. OpenAPI and .NET use the toolchain profile's bounded lingering-writer removal/backoff plus tracked-file restore; generated-code, Web and charts use standard exact-child removal; GSD uses workspace-only cleanup. The direct behavioural contract exercises invalid addressing, every profile, retry bounds, signal-driven retries, persistent failure aggregation and failed-status reporting, so workflow validators only need to prove that each lane delegates to the shared implementation. Juniper budgets are sized from measured shared-host runs plus the mandatory residue cleanup, which must never be the step that gets cancelled: generated-code 15 minutes, OpenAPI 25, web 25 and GSD-state 5; the hosted fork counterparts retain 5, 10, 15 and 2. The PR .NET unit suite is split into two parallel jobs per route (#4127). SyRF.ProjectManagement.Mongo.Data.Tests alone (about 2,500 replica-set tests) takes 25–33 minutes on CI, so the single 45-minute job was being cancelled at its limit. test-dotnet now runs every other test project (it still builds the whole solution, and tests a throwaway copy of syrf.sln with that one project removed so new test projects are picked up automatically) with a 35-minute budget: without Mongo.Data it takes about 13 minutes on most listeners but 21–23 on the slower Pomegranate ones, whose solution build alone takes about 8; test-dotnet-mongo-data runs only the Mongo.Data project with a 50-minute budget. The hosted fork counterparts test-dotnet-fork and test-dotnet-mongo-data-fork get 15 and 50 minutes. test-summary reports the .NET route as passed only when both jobs of the selected route succeeded, and the SonarCloud jobs wait for both and download every dotnet-coverage* artifact; see the workflow README. Every one of these is pinned exactly (not at most) in test-pr-validation-gates-runner-routing.sh, test-pr-web-runner-routing.sh and test-pr-dotnet-runner-routing.sh, each budget declared once in its script so the assertion and its negative fixtures cannot drift, and each carrying fixtures one minute either side, so silently narrowing a budget fails CI rather than quietly reintroducing the cancelled-cleanup failure. The generated-code validator isolates its pnpm store; the OpenAPI validator isolates .NET/NuGet, pnpm, and temporary state; and the chart validator isolates Helm state. The Juniper web job also runs src/services/web/test/ci-nginx-test.sh (a Docker consumer on the rootful host, gated by web_nginx_changed, run as a plain step alongside the residue cleanup): it builds one throwaway nginx image and runs three containers on one network, labels every object syrf.ci.owner=<PR_WEB_JOB_KEY>, and removes them from an EXIT trap that also fires on SIGINT/SIGTERM, so a cancelled job does not leak them; test-pr-web-runner-routing.sh pins the step on both routes. The Juniper web job retains the 25-minute/4-GiB-per-process/two-worker caps, disables setup-node's automatic package-manager cache, and isolates pnpm state under a run-unique runner.temp path. Both Juniper .NET jobs use a four-processor cap and run the shared toolchain cleanup under distinct run-unique keys (syrf-pr-dotnet-… and syrf-pr-dotnet-mongo-data-…); test-dotnet verifies Helm and test-dotnet-mongo-data verifies Docker; both isolate .NET CLI plus NuGet global-package, HTTP, scratch and plugin-cache state under a run-unique runner.temp root; Testcontainers cleanup remains Ryuk plus the host's scoped age sweeper, never job-local or blanket Docker pruning. Workflow/route-contract changes must trigger every affected real lane. The existing twelve-listener pool and its per-listener CPU/memory/task limits are the concurrency boundary; do not add a constant Actions concurrency group because GitHub retains only one pending job per group and replaces older queued work. The reviewed sonar-api, sonar-pm, sonar-quartz, and sonar-identity routes use Pomegranate as documented below; every other Sonar, token, container-integration, publish, registry, GitOps, deploy, production, and privileged PR job remains hosted. Mutually exclusive token-bearing route consumers must use !cancelled() rather than always() so obsolete jobs stop. The shared PR cleanup helper is repository-owned, so checkout is intentionally the first step in all seven of those lanes and the routing contract rejects any command inserted before it. If checkout itself fails after partially modifying the workspace, the helper may not exist; the separately provisioned root-owned Juniper ACTIONS_RUNNER_HOOK_JOB_COMPLETED is the authoritative backstop. It purges the listener _work, temp and cache roots, proves them empty, and retains the fail-closed active-job marker when proof fails. This host control is verified by camaradesuk/server-config, not duplicated in SyRF workflow code.
  • The Preview entry routes check-label-juniper and detect-changes-juniper follow the same accepted same-repository boundary. Fork PRs use the explicit GitHub-hosted label fallback; comment events bypass both label routes and go directly to the independent /reseed-db handler. The no-checkout route selectors remain hosted. The workflow has no workflow_dispatch trigger: a selected older ref can execute that ref's older workflow implementation, so a default-branch no-op guard is not a sufficient safety boundary. There is no hosted detection fallback: it could only have run for fork heads, which must not reach the build chain — every job below it pushes to GHCR and reads repository secrets that fork pull_request runs are never granted. It was removed rather than left as a dead arm in the detect-changes fan-in. Fork previews therefore require a separately authorized promotion path rather than the shared build chain. For the same reason the always()-guarded publishing tail (retag-unchanged, write-versions, update-pr-status, create-tags) must gate on needs.detect-changes.result == 'success': always() is broader than !cancelled(), and a skipped detection reaches those jobs with every *_changed output empty, which reads as "nothing changed" and copies :latest onto the PR's head SHA before shipping those tags to cluster-gitops. .github/scripts/test-preview-entry-gates-runner-routing.sh pins the publishing-tail guards plus the fork exclusions on the secret-bearing metadata refresh and write-token label repair. Both Juniper jobs are read-only, have non-persisted checkout credentials, bounded timeouts, run-unique state, and always() cleanup whose root is derived from job-level keys even when preparation fails. The label gate cannot mutate pull requests; its rare lock-db/use-snapshot repair is handed to a separate no-checkout hosted job with only pull-requests: write. No Preview build, package, registry, tag, GitOps, deployment, reseed, cleanup, or other credentialed job is moved by this entry-gate carve-out.
  • The default-false run_trusted_tests_only dispatch input selects the trusted test lanes — test-web and the two halves of the .NET unit suite, test-dotnet and test-dotnet-mongo-data — all self-hosted on juniper-ci. The canary allowlist and the main-workflow self-hosted allowlist happen to coincide again as of 2026-08-11, but they are separately defined and have diverged before (between 2026-08-07 and 2026-08-11 test-dotnet was canary-allowed while GitHub-hosted). Keep asserting them independently; do not collapse one into the other. Apart from detect-changes and workflow-summary orchestration, every other job must have a top-level AND deny guard for the canary.
  • The token-bearing sonar-api, sonar-pm, sonar-quartz, and sonar-identity PR consumers run on the persistent Pomegranate capacity in juniper-ci, requesting [self-hosted, linux, x64, juniper-normal-ci, pomegranate-ci]. They remain same-repository-only, retain exactly contents: read, check out full history first with persisted credentials disabled, use action-managed JDK 21 and .NET 10 (with .NET 8 additionally retained for Quartz), cap .NET at four processors, isolate .NET CLI/NuGet/temp state under a run-unique runner.temp root, and delegate always() cleanup to the shared toolchain profile. Their 25-minute budgets, exact service-token references, route labels, trigger gates, and first-step/cleanup contracts are pinned by test-pr-pomegranate-sonar-runner-routing.sh; changes to that contract force all four real scans. Optional persistent Sonar cache optimisation is not a routing prerequisite. All remaining Sonar jobs stay GitHub-hosted. Forks cannot launch any token-bearing scan.
  • The heavy e2e-tests.yml/e2e-tests job is the sole consumer of the two dedicated E2E listeners in juniper-ci, pomegranate-01 and the Bramble guest bramble-e2e-vm-01 (provisioned by camaradesuk/server-config roles/bramble_e2e_vm), requesting exactly [self-hosted, linux, x64, ${{ matrix.runner }}] where each matrix entry names its host-specific label, syrf-e2e-pomegranate-01 or syrf-e2e-bramble-vm-01. It never requests the generic syrf-e2e label: pomegranate-02 and juniper-02 carry it too and have no E2E lifecycle hooks, so the pre-flight would fail there. The allow-list (runner name, label, hook paths, runner temp and workspace roots) is e2e/scripts/ci-listeners.json; its lightweight label gate, PR comment, and label-cleanup jobs use the ordinary Juniper route, and so does merge-e2e-reports, a read-only, no-secrets job on the ordinary label set that downloads the shard artifacts and runs the base-branch merger (it never checks out PR-head code). The full suite runs as four matrix entries two at a time (max-parallel: 2), one per host: both functional shards on bramble-e2e-vm-01, perf and Bulk PDF on pomegranate-01; smoke runs on bramble-e2e-vm-01 (its traces on pomegranate-01 show Chromium ERR_NETWORK_CHANGED aborts from sibling runners' Docker churn, #3423), and so, since 2026-10-15, does the bulk-pdf-only suite (label, Bulk PDF path changes or dispatch). Its stack (clamd 1.5 CPU/3 GiB in place of SQL Server, 3.45 of the four-core Compose allowance) fits the guest's 6 vCPUs and 24 GiB, it was the most frequent pomegranate-01 entry (21 runs, median 10 minutes, 2026-10-07 to 2026-10-15), and pomegranate-01 shares its host with the ordinary pool. In the full suite Bulk PDF stays on pomegranate-01: the Bramble side already carries the longer share (functional-1 plus functional-2, medians 15.9 + 11.8 minutes, against perf plus Bulk PDF, 9.6 + 10.3), so moving it there would lengthen the run. A single bulk-pdf entry cannot float between the two hosts without a shared label, and the only shared one, syrf-e2e, is forbidden below. Every pomegranate-01 entry, and only those, sets browser_netns (exported as E2E_BROWSER_NETNS=1), which runs Chromium in its own unprivileged user and network namespace with the stack's loopback ports bridged in (e2e/scripts/chromium-netns.sh), so the host's veth churn never reaches it; the Bramble guest restricts unprivileged user namespaces and runs the default launch. Each entry is its own 45-minute job. Its PR event uses pull_request_target, so the immutable base-branch workflow denies fork heads before explicitly checking out the accepted same-repository head SHA. Manual Docker-lane dispatch is permitted only when the selected workflow SHA is the repository's exact current protected default-branch tip; the job has only contents: read and checks out with persisted credentials disabled. Changes to the E2E workflow or its routing contract force a live PR smoke and a full main run. Before checkout it proves that $RUNNER_NAME is an allow-listed listener (unknown runners fail closed), that it is the host its entry's matrix.runner names, and that the listener's exact root-owned lifecycle hooks are installed, Docker/Compose/exec, availability of its allocated ports and run-scoped container names, and clean labelled Docker state; Chromium system dependencies are host-provisioned and the job installs the browser without sudo, while awscli-local is isolated beneath run-unique runner temp. The existing per-PR/main concurrency group and 45-minute timeout remain binding; superseded PR runs are cancelled, while a main push leaves the in-progress main run to finish and only replaces the single pending one. Each stack's ports, Compose project, containers, owner label (repo:run:attempt:e2e-tests:<shard>) and job root derive from its run-scoped E2E_STACK_ID (<run_id>-<run_attempt>-<shard>) and are exclusive to that stack, so no step names a fixed port or syrf-e2e name (.github/scripts/test-e2e-stack-identity.sh); all Compose publications and host .NET listeners bind to 127.0.0.1; Compose services and LocalStack Lambda executors have no-swap CPU/memory/PID limits aggregating to the listener's four-core allowance. Services run in isolated process groups, and same-job always() teardown removes those groups plus the exact Compose/Lambda objects; the root-owned completion hook is the cancellation backstop for labelled containers, network, volumes, reservations, PID files, checkout, temp, and cache residue. Resource profiling aggregates the runner and job-owned Docker cgroups, measures only exact job container/volume mutable bytes, and separately reports new exact job-referenced image IDs by virtual size; daemon-wide image totals are prohibited. .github/scripts/test-e2e-concurrency.sh pins this routing, trust, prerequisite, cleanup, and reporting contract; fork or untrusted code must never reach this Docker-socket lane.
  • The test-dotnet and test-dotnet-mongo-data unit lanes may start their existing untagged Testcontainers fixtures on Juniper (the replica-set fixtures run in the latter). The individually approved trusted-main test-dotnet-integration lane uses the same shared-Docker ownership contract: Ryuk stays enabled as the primary cleanup mechanism and the host runs an hourly, two-hour-age, org.testcontainers=true-scoped backstop sweeper; neither job may prune Docker or perform blanket container cleanup. The integration lane is excluded from run_trusted_tests_only, retains its exactly pinned 45-minute timeout and four-processor cap, serializes project execution, verifies Docker before test work, isolates .NET CLI/NuGet/temp state beneath a run-unique runner.temp root, and removes/verifies only its job-owned workspace and temp residue in an always() step. Every integration-reached Testcontainers builder applies an owner label; when SYRF_TEST_JOB_KEY identifies a CI-owned container it also applies 2-CPU/no-swap/256-PID limits and a 2-GiB memory cap (3 GiB and a 1024-PID limit for SQL Server, which spawns a worker thread per host logical processor at boot regardless of the CPU quota; 256 killed it on the Pomegranate pool), while local Testcontainers retain their previous resource behaviour. Every container the policy configures, local or CI, also gets a 65536 nofile ulimit: Docker's default of 1024 let WiredTiger exhaust its file handles in the shared Mongo fixture (see run-tests). The legacy raw-Docker Quartz fixture preserves its stable port/name, reuse and unless-stopped policy for local development, but in CI uses a sanitized run-unique name, the same owner/sweeper labels and SQL Server caps, --restart no, an atomically allocated loopback-only port, and removes that exact job-owned container in finally. Deploy, release, promotion, preview, token, publish, registry, GitOps, production, and privileged-PR jobs remain GitHub-hosted except for the four reviewed Pomegranate Sonar routes above. Every further move still requires its own review, the audited-listener isolation controls, all three controls above (audited-listener isolation, trusted triggers, least privilege), an update to the bullet that names it, and an update to EXPECTED_GLOBAL_SELF_HOSTED_ROUTES in .github/scripts/validate-workflows.sh. Do not add a SyRF-specific runner group or routing label; use the shared juniper-ci group and org-wide juniper-normal-ci label, adding the org-wide pomegranate-ci label only for reviewed Pomegranate capacity.
  • preview-expiry.yml:expire (added 2026-09-22) uses juniper-ci with [self-hosted, linux, x64, juniper-normal-ci]. It has no pull_request trigger at all (schedule + workflow_dispatch only), so no fork-originated code can ever reach it; it needs no undeclared host tool (gh and jq, both already relied on by pr-preview.yml's Juniper routes, plus a pinned actions/setup-node for the selection script); and its permissions are least-privilege for the job (pull-requests: write to remove the preview label, issues: write to post the explanation comment, contents: read). It reads and mutates only same-repository PR metadata (labels, comments) it selects itself from gh pr list --label preview, never PR head code. Its expire job delegates always() cleanup to .github/scripts/juniper-job-cleanup.sh workspace expire-previews, added alongside check-gsd-state/validate-pdf-path-case-folding as another no-RUNNER_TEMP-artifact workspace-profile lane. The workflow's second job, cleanup, has no runs-on of its own — it is a matrix of uses: calls into the reusable _preview-cleanup.yml (one per expired PR, secrets: inherit), which stays on ubuntu-latest exactly like the cleanup-tags job it was extracted from in pr-preview.yml, so neither job is a self-hosted route.
  • statistics-staging-preflight.yml:preflight (added 2026-09-29, FEAT-024) uses juniper-ci with [self-hosted, linux, x64, juniper-normal-ci]. It is workflow_dispatch-only and its job runs only for refs/heads/main, so no fork-originated or unreviewed branch code reaches it. Workflow and job permissions are contents: read; the only credential is a GitHub App token minted with repositories: cluster-gitops and permission-contents: read (the pr-preview.yml read-token pattern), used for GitHub contents API reads. Its other reads are anonymous HTTPS GETs against staging. It needs only the host python3 (standard library), writes a JSON capture inside the workspace, uploads it as an artifact and delegates always() cleanup to juniper-job-cleanup.sh workspace statistics-preflight. Since 2026-09-30 (syrf #3806) the job also has id-token: write and impersonates syrf-stats-preflight@camarades-net.iam.gserviceaccount.com, whose Workload Identity binding admits only repo:camaradesuk/syrf:ref:refs/heads/main and which holds roles/container.clusterViewer plus the namespace view role, to run kubectl get on the staging Deployments and pods. It installs a pinned, checksum-verified kubectl into RUNNER_TEMP when the host lacks one and uses get-gke-credentials (token kubeconfig), so it needs neither gcloud nor gke-gcloud-auth-plugin on the host, the same pattern as pr-preview.yml. The raw kubectl JSON stays in RUNNER_TEMP and is deleted by an always() step.
  • statistics-staging-preflight.yml:parity (added 2026-09-30, FEAT-024) uses the same group and labels with contents: read only. It needs no OIDC or cluster access: it calls the staging Identity and API over HTTPS with SYRF_STATISTICS_EVIDENCE_CLIENT_SECRET (scope statistics:parity:read, one read-only route). Since syrf #3908 it is main-only (github.ref == 'refs/heads/main'; there is no branch dispatch with project_ids any more) and declares environment: statistics-evidence: the secret is an environment secret of that environment, whose deployment-branch policy admits main only, not a repository secret, so no branch's workflow code can read it. Scheduled runs use refs/heads/main and are admitted. The preflight job deliberately has no environment, because one would rewrite its OIDC subject away from the ref:refs/heads/main binding. Host python3 only; always() cleanup through juniper-job-cleanup.sh workspace statistics-preflight.
  • statistics-operator.yml:operate (added 2026-10-02, FEAT-024, syrf #3908) uses the same group and labels with contents: read only and is pinned in EXPECTED_GLOBAL_SELF_HOSTED_ROUTES. It is workflow_dispatch only (inputs environment (staging only), operation, project_id (validated GUID), family, confirm, rollout_complete_confirmed), runs only when github.ref == 'refs/heads/main', and declares environment: statistics-operator, whose main-only deployment-branch policy releases the environment secret SYRF_STATISTICS_OPERATOR_CLIENT_SECRET (scope statistics:operate: the statistics pending-index, fold, backfill/rebuild and parity routes only). Every state-changing operation requires confirm, checked in the workflow and again by scripts/statistics-operator.py; the access token is masked before use and the job summary carries only sanitised statistics metadata. Host python3 only; always() cleanup through juniper-job-cleanup.sh workspace statistics-operator. validate-workflows.sh pins both statistics environments, the preflight job's lack of one, the main-ref condition on both secret-bearing jobs, and that each secret is referenced only by its own workflow.
  • test-statistics-scripts.yml:unit (added 2026-09-29) runs the offline FEAT-024 statistics script unit tests on juniper-ci (the Bramble native route since 2026-10-15) for push to main and same-repository pull requests; fork heads use the hosted unit-fork lane. It has contents: read, non-persisted checkout credentials, host python3 only, and always() cleanup through juniper-job-cleanup.sh workspace test-statistics-scripts.
  • test-dotnet was rolled back to ubuntu-latest on 2026-08-07 and restored to juniper-ci on 2026-08-11 once the root cause was fixed: SyRF.Identity.Endpoint.Tests invokes helm as a bare process name, which GitHub-hosted images ship and the Juniper hosts did not. Helm v3.21.3 is installed system-wide by the helm role in camaradesuk/server-config#156, and the lane records helm version --short before restore/build/test work. Before routing any job self-hosted, check for undeclared host-tool dependencies (helm, crane, docker, cloud CLIs): a bare Process.Start/run: invocation of a tool the workflow never installs is exactly this trap. A change to the main workflow must itself trigger the affected lane so routing changes cannot merge without live evidence.
  • The two Claude agent jobs (claude.yml, claude-code-review.yml) remain on juniper-ci. The review workflow is one direct comment-driven job with in-job authorization, the existing 60-minute timeout, all eleven specialist review areas and native action progress/inline comments. There are no label controllers, status publishers or cancellation jobs. claude.yml retains its exact per-event trusted-author predicate and general behavior. .github/scripts/test-claude-review-comment.cjs exercises the comment authorization and no-request paths; the timeout contract and global routing allowlist pin the remaining routes. A skipped action still fails the requested job, but review runs publish no synthetic required status. Before rollout, remove only obsolete claude-review/policy requirements, preserving test checks, approvals and conversation resolution. See Claude comment reviews and migration.
  • Active JavaScript actions use Node 24-compatible releases. When updating an action, retain its existing workflow semantics and choose a Node 24-compatible release; immutable pins must identify the release they pin. Archived workflows are historical reference only and are not part of this runtime baseline. Run .github/scripts/validate-workflows.sh after workflow changes; it enforces the checkout contract used on Juniper.
  • Preview version calculation is consolidated through _preview-gitversion.yml: one read-only GitHub-hosted job checks out the exact PR head with full history, runs GitVersion once for each changed service configuration (including Lambda-only S3 Notifier changes outside the Docker matrix), and forwards the existing output contract plus per-service outcomes. Web artifacts, each Docker matrix entry, and S3 Notifier packaging use only their matching successful outcome, so successful siblings may stage images or packages after an unrelated calculation fails. Those staged artifacts are not activated: write-versions, git tags, and deployment progression remain fail-closed on aggregate version success and required build prerequisites; write-versions may accept a skipped version job only when change detection proves that no service changed. Status output must identify per-service failures and a staged-but-blocked Lambda package rather than claiming activation. test-ci-cd.yml provides a manual version-only parity canary that independently recalculates all eight configs, fails on any consumed-output mismatch, and cannot build, publish, deploy, promote, or mutate an environment.
  • validate-workflows.yml deliberately has four root execution lanes plus a tokenless summary. actionlint stays isolated because reviewdog needs pull-requests: write; script-tests and custom-validation stay in separate contents: read jobs because they execute PR-controlled repository scripts and therefore require independent fresh checkouts and environments. Only shellcheck, yamllint, inline workflow-security scans, and inline services-JSON validation share the consolidated passive-validation job. The final summary has permissions: {}, depends on exactly those four root jobs, reports the four passive outputs plus the two executable-job results, and fails unless every dependency succeeds.
  • Passive validation is collect-all but fail-closed. Each passive leaf check uses continue-on-error, and evaluator steps convert every failed or skipped leaf into the corresponding published failure before final passive aggregation exits non-zero. The final aggregation itself must never use continue-on-error. A failed yamllint install/configuration fails yamllint while unrelated shell, security, and JSON validators still execute. The hardcoded-secret and dangerous-command pattern findings preserve their pre-consolidation advisory-only contract: they emit warnings but do not fail CI; the workflow_security output and the summary's Security advisory execution row report whether those checks executed successfully, not whether they found advisory matches. Missing .github/services.json is deliberately fail-closed: a change that removes this required validation input must update the validation design and deterministic fixture rather than passing as "nothing to validate". Do not rejoin a PR-controlled executable validator or fixture to the passive job; the consolidation contract pins the job graph, separate checkouts, least privilege, non-persisted credentials, passive outputs, summary fan-in, and an executable-rejoin negative fixture. In script-tests, yq verification and the consolidation contract must run immediately after checkout, before any mutable repository fixture; the independent custom validator pins that prefix, and the consolidation contract pins both authoritative launcher bodies and rejects disabling conditions.

E2E test deadline and teardown reserve

The dedicated E2E job retains its 45-minute deadline. Immediately before Playwright, e2e-test-budget.sh subtracts elapsed setup, a 17-minute post-test reserve and a 30-second forced-stop grace. Exhausted setup budgets fail the test step without launching Playwright; timeout and command failures retain the failed test outcome. Post-test steps have explicit limits totalling 16 minutes, leaving one minute for step transitions. Scoped service and Docker cleanup runs before artifact uploads, so a slow upload cannot consume the teardown allocation. Budget fixtures include slow setup, the exhaustion boundary, invalid clocks and the post-test timeout sum.

The E2E lane additionally has actions: read to prove that a resource's owning run attempt has completed before recovering it. Host identity and hook-path checks still precede checkout; ownership recovery and fixed-port proofs run after checkout so their implementation can be tested as repository code. See the CI recovery contract.