[anth-volk]

Fixes #1067

A run of a UK-wide or nation report on the release's local-area dataset (the dataset UK constituency and local-authority regions use, microcosm_uk_2024_25_dense on .py 6.17.1) now calculates its full results, cliffs included when asked, together with its constituency and local-authority results and the codes of the areas disclosure control withholds, and stores constituency_impacts, local_authority_impacts and withheld_area_codes rows. It is an ordinary report run, started by run ID; no field marks it, its simulations' dataset does.

Deploy order: the Public API's withheld_area_codes table and its grants (PolicyEngine/policyengine-api#4305, revision 1e1e3f079f2c) must be live in production before sim's release. They are: #4305 merged on 2026-11-13 as ca4d396e, and the Public API's release run 43519970359 applied the revision and its grants, and verified them, in staging and then in production (2026-11-13 23:20 UTC). Without them, a run of such a report would fail at its completion's insert; no other report writes to that table. Nothing creates such reports yet: the simulation service's data_source option comes in a later PR, which reads this PR's manifest declaration.

What changes

• UK worker (stage12_runtime/simulation.py, new stage12_runtime/local_area_datasets.py): it accepts a uk or country/* simulation whose provenance names the local-area dataset, which it refused before ("simulation provenance differs from the installed bundle"). It selects that dataset in place of the region's own, keeps the region's scoping (a nation's filter), and checks every area column the area results read as the dataset loads, before simulating. The rule is .py's routing rule (uk_area_regions_routed, routed_uk_area_dataset_name), the one the national aggregation's uk_area_breakdowns_omitted already uses. It calls none of the v1 executor's local-area code.
• Output plan (output_planning.py): a UK-wide or nation simulation on that dataset also copies UK_AREA_DATASET_COLUMNS, whether planned for a report or alone.
• Aggregation (aggregation.py, SimulationOutputBuilder.uk_withheld_area_codes): for a report on that dataset, the existing national aggregation stops leaving the constituency and local-authority results out, and adds each level's withheld codes from the same disclosure pass. Such a run fails if a level has neither published nor withheld areas, so a succeeded run always has rows for each level. Every other report's output, and its comparison with production, is unchanged.
• Economy output (SingleYearMacroOutput): optional constituency_withheld_codes and local_authority_withheld_codes, left out of the JSON when null, so every other result is written exactly as before. The gateway's golden OpenAPI spec gains the two fields.
• Stage 12 writes (report_run_results.py): the withheld_area_codes table model, in the Public API's shape, and one row per withheld code, keyed by the run and its two simulation runs, written in the completion's transaction. A result that lists none writes none, so national runs never touch the table.
• v2 version manifest (stage12_manifest.py, new stage12_manifest_models.py, the publisher): the UK entry declares local_area_reports (input_version, dataset), the dataset by the same routing rule, so the simulation service, which can't read the release kind from the catalog, can tell whether such a report is possible. Readers released before the field forbid unknown fields, and publish_stage12_v2_manifest can run before Cloud Run traffic moves, so the field is written only under a new key, active-schema-2-with-local-area-reports, which current readers read first while active-schema-2 holds its copy; active-schema-2 and the schema version 1 copies leave it out. The publisher requires both schema version 2 keys. The manifest's models moved, unchanged apart from the new one, to stage12_manifest_models.py (re-exported), keeping both files under the length limit. The Stage 12 doc's manifest section describes the key, the rollback handling, and that deleting active-schema-2 before a rollback also stops readers using the new key.

The design decisions it follows

From the local-area design's decisions (recorded 2026-11-13 and 2026-11-14):

• regions: uk and the four country/* nations, with the executor's routing rule deciding which releases have a local-area dataset (decision 3, reworded; item 12 lets nations take it);
• the local-area report always uses the release's local-area dataset (decision 4); with data_source, it is an ordinary report on that dataset returning its full results (the data_source design, replacing decisions 6, 8, 9 and 12);
• cliffs are calculated when asked (decision 5);
• withheld codes go in a new withheld_area_codes table (decision 7);
• Stage 12's existing national aggregation calculates the areas, with no separate module (decision 14);
• the Wales staging check (decision 13) belongs to the simulation service's data_source change; the area check above lets it read only the run's status.

Environment variables and secrets

None added. What these runs read, all already attached to coordinate_report and run_single_simulation_uk in v2_app.py:

• the Hugging Face token, from the Modal secret named by HF_MODAL_SECRET_NAME (required_keys declares it), in worker_secrets: the worker downloads the private local-area dataset with it;
• GOOGLE_APPLICATION_CREDENTIALS_JSON, from the Modal secret stage12-evaluation-gcp-credentials, in worker_secrets: output files and the constituency lookup file. That secret declares no required_keys, so the key it holds is unverified;
• MODAL_WORKER_DATABASE_URL, from policyengine-modal-worker-database (required_keys), in report_run_writer_secrets.

The manifest publisher reads no new variable; it reads only the packaged .py manifest, and its job still needs only MODAL_TOKEN_ID and MODAL_TOKEN_SECRET. No workflow changed.

Tests

Each new test was seen failing for the expected reason first; only new and modified tests were run, by node ID.

• Contract: the manifest declaration under its own key, never seen by older readers, ignored after a rollback, only on an economy-capable UK entry, naming a bundle dataset; the region-dataset rollback step sending every reader to active-with-cache.

• Executor: the worker accepting, selecting and checking the local-area dataset for uk and nations, refusing it on a release that doesn't route and at another revision, and over the installed .py bundle unpatched; the recognition rule; the plan's area columns and cliffs; the aggregation's areas, withheld codes and area check, for UK-wide, nation, constituency, local-authority and national-dataset reports; the publisher's declaration and its two required keys (including --verify-plan with active-schema-2 deleted).

• Stage 12 persistence: withheld rows from a result, and, on PostgreSQL 17 built from the Public API at #4305's revision with its privileges applied, writing them as the Modal worker role beside the area rows, no second set on a repeated completion, and a failed withheld insert rolling the whole completion back.

• The gateway's golden OpenAPI spec, regenerated for the two fields.

• The economy output's schema tests: SingleYearMacroOutput's fields and its published OpenAPI schema are the current result keys plus the two withheld-code fields (LOCAL_AREA_WITHHELD_CODE_KEYS); dumped and built results still have the current keys alone.

libs/policyengine-simulation-persistence/public-api-commit now points at #4305's merge commit, ca4d396e, for the Public API schema job, which compares only policyengine-simulation-persistence's tables, so it doesn't check Stage 12's withheld_area_codes model. Since the previous pin the Public API added no other migration.

Not covered by CI: no CI job sets the REPORT_RUN_TEST_* URLs, so Stage 12's PostgreSQL tests skip there (as before this PR); the three new ones were run by hand.

Unverified

• The coordinator's memory (2 CPU, 8 GiB) for a UK-wide report on the local-area dataset, which decodes both full output files and runs the national analysis and 200 disclosure replicates: needs a measured Modal or staging run.
• The v2 UK image holds only the UK default dataset (policyengine bundle install installs one per country), so each new worker container downloads the local-area dataset first, about 90 s in production.
• That the local-area dataset gives Northern Ireland households non-empty area codes, so a Northern Ireland report passes the area check: not checked against the dataset itself.

Reviews

Six read-only review rounds. Round 1 found one bug, fixed test-first: the documented rollback step (deleting active-schema-2 and active-with-region-datasets) didn't stop readers released with the new key from reading it. Rounds 2 to 6 found none; their code-quality findings are fixed. Round 6 covered the two CI test-expectation fixes and the pin.

🤖 Generated with Claude Code