[anth-volk]

Fixes #1071

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Changelog

• This PR adds a changelog fragment, changelog.d/c1-26-shared-simulation-runs.changed.md, whose type is breaking, added, changed, fixed or removed.

Summary

Stage 12's v2 functions can now share simulation runs between report runs, as the Public API's simulation service is to do for identical requests. This follows the migration's shared-run decisions: answer 4 (each run stores what its own simulation decides), answer 5 (the coordinator waits for a run another call holds), answer 8 (household reports are keyed too, so the household function gets the same treatment), interim A (report runs asking for cliffs are refused for now; cliffs come back in a later stage as a calculation of each simulation run), and answer 3 with interim B (forced re-runs, and for now re-runs without a key, calculate extra runs nobody joins; a test covers this, and it needs no code).

Stacked on #1068, the local-area report profile; its commits up to 45ffb33 are merged in.

Waiting instead of cancelling (stage12_runtime.shared_simulation_runs, report_run_coordination):

• The coordinator starts a worker for every simulation run that is pending with no Modal call, whichever report run created it. The worker's claim lets only one call hold the run.
• For a run another call holds, the coordinator probes that call without cancelling or releasing it. Once the call ends, it reads the run again:
  • a run that succeeded is read from its stored file and checked as a reused run is, dataset and region included;
  • a run that failed fails the report run with that run's recorded failure;
  • a run left pending with no call gets this coordinator's worker;
  • a run still unfinished under the call that ended is failed while it still records that call (fail_simulation_run_held_by, modal_call_ended), so every report run waiting for it stops.
• A probe that raises reads the run first. It counts as the call ending only once Modal reports the call ended (modal_call_state), because Modal's client also raises for a lost connection while the call goes on.
• A coordinator that fails, is cancelled or is pre-empted writes only its report run. It never cancels workers or fails simulation runs other report runs may share. Its own worker's run is failed only for that worker's own failure, while the worker's call holds it. release_simulation_run is removed.

The household function (household_report_recording):

• It calculates the sides it can claim first.
• It then waits for a side another call holds, with the same rules, for up to 780 seconds from its start (HOUSEHOLD_SIDE_WAIT_SECONDS, derived from its 900-second Modal timeout less two minutes to finish). The holding call is another household function, which goes on after recording the run, so the wait reads the run before each probe and takes a finished run at once. A side still held when the window closes fails the report run with household_simulation_run_unavailable, and the run is left to its call.
• When it fails, it fails only runs it holds.

One simulation run on both sides:

• The contract accepts a report run naming one simulation run as baseline and reform when both sides are the same simulation, policy included. That case covers the staging check of current law against current law.
• The coordinator and the household function calculate or read that run once.
• The result rows store its poverty and inequality rows once.

What a simulation run stores (output_planning.resolve_simulation_output_plan):

• Each worker gets its own simulation's plan, the same whatever report names the run:
  • the labour-supply response columns only when its own policy turns a response on;
  • in the US, the two inputs .py's labour-supply output reads from either side;
  • never the cliff columns, so a US national run keeps its region-group split.
• The report keeps its pair's plan for the aggregation. A side storing no responses the report reads is read with them as zeros.
• A run that succeeded before this change is reused when its file holds what the report reads from that side, and refused with reused_simulation_output_incompatible otherwise.
• A report run asking for cliffs fails before any worker starts, with cliff_impacts_unavailable, and touches none of its simulation runs.

Precomputed US baselines are planned the same way, so a report run's current-law US national baseline finds them. The first deployment computes the 60 baseline files again under their new cache identities. A comparison-path call (a shadow comparison with a production job, or the operator-only POST /internal/stage12/reports) still plans its baseline with its report's plan, so its US national baseline no longer reads the precomputed files and calculates all 20 region groups. This costs compute, not correctness.

Stuck-run sweep:

• The 24-hour age limit now fails a simulation run that records a Modal call, whatever report runs name it. New report runs could otherwise keep waiting for a call Modal never ends.
• A run no call has claimed, such as a budget window's later year, still waits for the report runs naming it.
• The pass that judges runs older than 10 minutes is unchanged.

For the simulation service (C1 · 27): it must not create shared pending runs, that is, name a run another report run created and that hasn't succeeded, until this is deployed to the default worker version's Modal application. Until then, a report run naming a run another worker holds would cancel that worker, and a failing report run would fail runs others wait for. Joining a run that already succeeded is safe today.

Deployment and environment:

• No change to workflows, images or Terraform, and no new environment variable or secret. v2_app.py changes only where the household functions' 900-second timeout comes from: it now imports HOUSEHOLD_FUNCTION_TIMEOUT_SECONDS from stage12_timeouts, which it already imported from.
• The coordinator and the household function now ask Modal for a call's state (modal_call_state) and restore calls by ID with the Modal client inside their containers, as the stuck-run sweep and the coordinator's restore already do. This is not yet verified in a deployed container.

Known limits:

• A waiting coordinator or household function holds one of its containers (25 coordinators, 20 household functions) while it waits.
• The sweep doesn't end waits: waiters fail a run whose holder died, once Modal reports that call ended.
• A probe or state lookup that hangs keeps a coordinator past its wait window until Modal's client gives up or the 5,400-second limit ends the call.

Testing

• Each change was tested first: a failing test was committed with "(failing)" in its subject, then the change.
• Tests were run individually by node ID: the new and changed executor, contract and persistence tests. The persistence tests ran against a disposable PostgreSQL 17 database that the Public API's migrations created.
• Ruff format and lint, the repository rules, the quality checks, the lock checks and Pyright passed before each commit.
• Four read-only review rounds, the last on the fixes after the third; their bugs were fixed test-first.

🤖 Generated with Claude Code