[anth-volk]

Fixes #4308

Summary

The release workflow's last job, "Clean up old Cloud Run images", has failed in every release run that reached it since the 2026-11-12 22:58 UTC release. Since ubuntu-latest became Ubuntu 26.04, each release pushes three manifests instead of one, and the cleanup deletes them in an order that Artifact Registry refuses. This PR makes the build push one manifest again and makes the cleanup count, keep and delete by release.

• .github/scripts/build_cloud_run_image.sh passes --provenance=false to docker build, in both the dry-run and the real invocation (now one shared command array), so each release pushes a single image manifest.

• .github/scripts/cleanup_cloud_run_images.sh lists tags (--include-tags) and counts a release as a tagged manifest:

  1. The newest KEEP (15) tagged manifests are the kept releases; older tagged manifests are deleted first. For the releases built on 26.04 so far, those are image indexes.
  2. Then it deletes the untagged manifests uploaded before the oldest kept release: the image and attestation manifests whose index was deleted in step 1 or by an earlier run. Untagged manifests listed with the same upload time as the oldest kept release, or a later one, are kept. That keeps the manifests a kept index references when they are listed with its upload time, as the 26.04 releases appear to be (see Evidence).
  3. A deletion that Artifact Registry refuses with referenced by parent manifests is logged as Kept <ref>: another manifest still references it. and not counted as a failure. That happens to a manifest uploaded just before the kept index that references it, normally while that index is the oldest kept release, and to a manifest whose untagged index (left when a tag is pushed again) is deleted later in the same run; the next run deletes that one.
  4. KEEP must be a whole number of 1 or more (checked before listing). Rows sort with LC_ALL=C, so manifests with the same upload time always sort in the same order.

  A failed listing or any other deletion failure still fails the job, DRY_RUN=1 still only prints, and the script still runs on bash 3.2.

• The script's header comment, the comment above the workflow step, and a changelog fragment (changelog.d/cloud-run-image-cleanup.fixed.md).

Evidence

Runner and build. The 2026-11-10 release (run 42898604898, df394c589) built on Image: ubuntu-24.04 (runner image ubuntu24/20261101.360, Docker 28.0.4). Its "Deploy and test staging Cloud Run candidate" log shows #15 writing image sha256:4ceca0b3…. From the 2026-11-11 release (run 43051821762, 8b906f8c0) the job runs on Image: ubuntu-26.04 (ubuntu26/20261111.196, Docker 29.4.2). Run 43489854522 (f9c619dd8) shows:

#15 exporting manifest sha256:c04a75b04b260675949fb1fe348e1cd0d4c19d9a523cf2cda513e14edb0e3235 done
#15 exporting config sha256:e65cc228618d14b6be8f455500c7cb2f01bc9793c873fb9578d40a30f8294ea8 done
#15 exporting attestation manifest sha256:7e89981b17b3f08f475515e59f9a91c251d98992f87b7dd6c63332cf7ef13115 done
#15 exporting manifest list sha256:7961eca22cff068b53c3ae69e0fe15ceeeaf2fbb8cb30195916a0abcda3501ce done
f9c619dd86edb31e5227677d5e4f7dc0763d6974: digest: sha256:7961eca22cff068b53c3ae69e0fe15ceeeaf2fbb8cb30195916a0abcda3501ce size: 856

Docker 29 stores images in containerd, which keeps the provenance attestation BuildKit attaches by default, so the push is an image index (tagged with the release commit) that references the image manifest and an attestation manifest (both untagged).

--provenance=false gives one manifest. Locally, Docker 29.6.2 with the containerd image store (driver-type io.containerd.snapshotter.v1) and docker build --platform linux/amd64 --metadata-file … of a FROM scratch image:

Flags	Build output	containerimage.descriptor.mediaType
(default)	exporting manifest, exporting attestation manifest, exporting manifest list	application/vnd.oci.image.index.v1+json
--provenance=false	exporting manifest only	application/vnd.oci.image.manifest.v1+json

No SBOM attestation was attached by default, so --sbom=false isn't needed. Nothing was pushed to Artifact Registry.

Cleanup counts. On 24.04 the cleanup listed 16 manifests and deleted 1 per release (run 42898604898). On 26.04 it listed 18 and deleted 3 (run 43051821762), so KEEP=15 kept about 5 releases. The failed runs:

Release run	Commit	Listed	To delete	Deleted	Refused child's parent index
43324691894	2eae02aeb	18	3	2	7dcceef4… (release cd513dacc)
43333854428	9b02ce57b	19	4	3	81744ed7… (release 3fa2de575)
43452183386	f843c3f5a	22	7	6	72b84d95… (release c4419dcea)
43489854522	f9c619dd8	19	4	2	0bbc9874… (release df9d51c64), twice
43519970359	d7042b226	20	5	3	1546b9e1… (release 2eae02aeb), twice

Each refusal reads like this one from run 43489854522:

ERROR: (gcloud.artifacts.docker.images.delete) manifest is referenced by parent manifests: failed precondition manifest has referenced parents: policyengine-api/manifests/sha256:0bbc987444b3c4b8d2e8fbe26d7e351088a46a2881741deb137f0e32692e5595

Working the counts through from the first 26.04 release, every manifest of the refused child's release was due for deletion in each of these runs. The order of the deletions caused the failures, not a 15-manifest limit falling inside a release. Each refused child's digest sorts above its index's digest (dddf4ca5 > 7dcceef4, e8294dd0 > 81744ed7, e13d0247 > 72b84d95, 77689f2f and 646094b7 > 0bbc9874, c30b7f0e and 984e8447 > 1546b9e1). That fits the three manifests of a release being listed with the same createTime: the old sort -r on whole rows then ordered them by digest. The old script didn't print createTime, so this is inferred, not observed. The new script doesn't depend on it: it deletes tagged manifests before untagged ones, and logs a refusal instead of failing. Deleting an index left its children in the registry, and later runs listed and deleted them.

The listing format comes from the Google Cloud SDK source (579.0.0, command_lib/artifacts/docker_util.py): for a repository, gcloud artifacts docker images list --include-tags returns createTime (the upload time), package, version (the digest) and tags. A local check with the SDK's own value[separator='|'] printer showed several tags joined with ; and an empty field for an untagged manifest. The job installs version: latest of the SDK.

Environment variables and secrets

No variable or secret is added. What the changed scripts read, and where each comes from:

cleanup_cloud_run_images.sh: job cleanup-cloud-run-images (environment: production), step "Prune old Cloud Run images", which has no job or step env:.

Variable	Provider	Wiring
KEEP, DRY_RUN, AR_LOCATION, AR_PROJECT, AR_REPO	Defaults in the script (15, 0, then the three CLOUD_RUN_* values below)	Not set by push.yml. No repository or environment variable of these names exists, and none would reach the step without an env: entry.
CLOUD_RUN_REGION, CLOUD_RUN_PROJECT, CLOUD_RUN_ARTIFACT_REPOSITORY	Defaults in cloud_run_env.sh (us-central1, policyengine-api, policyengine-api)	The job sets none of them, so the cleanup prunes the repository the build pushes to.
Other CLOUD_RUN_* and V2_* names in cloud_run_set_defaults	Defaults in cloud_run_env.sh; V2_SUPABASE_PROJECT_REF and V2_SUPABASE_ENVIRONMENT are only exported	Not used by the cleanup.
GITHUB_SHA, GITHUB_RUN_NUMBER	GitHub Actions default environment variables	Read by cloud_run_set_defaults; not used by the cleanup.
gcloud credentials and project (CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE, GOOGLE_APPLICATION_CREDENTIALS, CLOUDSDK_CORE_PROJECT, …)	google-github-actions/auth@v2, from the GitHub Actions secrets GCP_WORKLOAD_IDENTITY_PROVIDER and GCP_DEPLOY_SERVICE_ACCOUNT	Both secrets are repository-level (none at the staging or production environment level), so this production-environment job gets them; the job has id-token: write. The auth step exports the credential variables to later steps, as run 43489854522's log shows for the cleanup step.

build_cloud_run_image.sh: job "Deploy and test staging Cloud Run candidate" (environment: staging), step "Build and push Cloud Run image".

Variable	Provider	Wiring
CLOUD_RUN_IMAGE_TAG, CLOUD_RUN_TAG	Outputs of the "Compute Cloud Run staging metadata" step (compute_cloud_run_metadata.sh, from GITHUB_SHA and GITHUB_RUN_NUMBER)	The step's env: sets both from steps.cloud_run.outputs.
CLOUD_RUN_IMAGE_PLATFORM, CLOUD_RUN_IMAGE_URI, CLOUD_RUN_IMAGE_NAME, CLOUD_RUN_REGION, CLOUD_RUN_PROJECT, CLOUD_RUN_ARTIFACT_REPOSITORY	Defaults in cloud_run_env.sh (linux/amd64, and the URI built from the others and the tag)	Not set by the job or the step. The job's env: sets CLOUD_RUN_SERVICE and runtime-cache settings, which the build doesn't use.
CLOUD_RUN_DRY_RUN	Default 0	Not set in the workflow; the tests set it.
gcloud and Docker credentials	Same repository-level secrets through google-github-actions/auth@v2; gcloud auth configure-docker sets up Docker	Unchanged.

Unverified: the cleanup's deletions need artifactregistry.versions.delete and artifactregistry.tags.delete, and its listing artifactregistry.dockerimages.list, on the deploy service account. I can't read the IAM policy from here. The same account listed and deleted manifests in the failed runs (2 deleted in run 43489854522), and --include-tags doesn't change the API call that the current SDK makes for a repository listing.

Testing

Only the tests this PR adds or changes, by node ID:

• tests/unit/test_workflow_guard_scripts.py: the 13 test_cleanup_* tests plus 4 cases of test_cleanup_rejects_a_keep_below_one_release. The fake gcloud works like Artifact Registry in two ways: it refuses to delete a child manifest while its parent index still exists, and it drops deleted manifests from later listings. The test releases mirror the failed runs: same upload time for the three manifests, and an attestation manifest that sorts ahead of its index. Two tests run the cleanup several times against one fake registry: a manifest refused because its untagged index still existed is deleted by the next run; and with KEEP=2, a manifest uploaded a second before its index gets no deletion request while that index is the newer kept release, a refused one while it is the oldest kept release, and is deleted after the index once the index is no longer kept.
• tests/unit/test_cloud_run_candidate_deploy.py: test_build_cloud_run_image_dry_run_turns_off_provenance_attestations, test_build_cloud_run_image_pushes_one_manifest_without_provenance (real path, fake gcloud and docker), and the two existing build tests.

The new tests were committed first and failed against the old scripts (11 of 13, then 4 of 4). All 21 pass now, and the cleanup tests also pass under macOS's /bin/bash 3.2.57. make quality passed in the pre-commit hook, and shellcheck, run the way the ci-scripts guard runs it, reports nothing for the changed scripts.

Not tested: the LC_ALL=C on the sort. It only fixes the order in which manifests with the same upload time are tried. A different order can at most leave a refused manifest for the next run, as in the two-run test. The fake listing prints its rows whatever the arguments, so the ;-joined tags and the createTime mapping rest on the SDK source and the printer check above, not on the tests.

After merging

The first release after this merges builds a single-manifest image. Its cleanup keeps all the releases in the registry (there are fewer than 15), deletes the image and attestation manifests left behind by the failed runs, and keeps the 26.04 releases' indexes and the manifests they reference until they age out of the newest 15.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (d7042b2) to head (1f6c23f).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4311   +/-   ##
=======================================
  Coverage   93.47%   93.47%           
=======================================
  Files         369      369           
  Lines       21455    21455           
  Branches     3400     3400           
=======================================
  Hits        20056    20056           
  Misses        816      816           
  Partials      583      583           

Flag	Coverage Δ
integration	56.62% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.