[anth-volk]

Fixes #4302

Summary

Revision 94dc43079186 (on 1e1e3f079f2c), autogenerated, lets the simulation service find each simulation and report by a key and join an existing run instead of creating new rows. It covers economy reports, household reports and budget windows.

• Keys. simulations and reports gain nullable key_version and definition_key, unique on the pair (uq_simulations_key_version_definition_key, uq_reports_key_version_definition_key).
  • reports also gains dataset_independent_key, which a national report and its local-area report share. It is indexed on its own (ix_reports_dataset_independent_key), because the app finds one report by the other's key, and the key holds its own key version.
  • Checks require 64 lowercase hexadecimal digits, a positive key_version, and every key column or none.
  • Existing rows stay unkeyed, and row IDs stay random.
• Forced re-runs. simulation_runs and report_runs gain forced, false by default. For a budget window, recalculate is the forced marker.
• One run per version set. Three partial unique indexes leave out failed and forced runs. So a failed run never counts, and a forced re-run is an extra run nothing joins:
  • uq_simulation_runs_unfailed_unforced_versions on (simulation_id, year, tax_benefit_model_version_id, country_package_version, policyengine_version) where status <> 'failed' AND NOT forced;
  • uq_report_runs_unfailed_unforced_versions on (report_id, country_package_version, policyengine_version) where budget_window_run_id IS NULL AND status <> 'failed' AND NOT forced;
  • uq_budget_window_runs_unfailed_unforced_versions on (report_id, tax_benefit_model_version_id, policyengine_version) where status <> 'failed' AND NOT recalculate. budget_window_runs gains policyengine_version for it; it is nullable, and unversioned window runs are never joined.
  • All three are declared for SQLite too, so the unit tests' databases keep the rule.
• Each user's requests. user_report_run_associations records each user's request for a report run:
  • the user (CASCADE) and the run (RESTRICT, indexed);
  • requested_at (required, no default);
  • trigger (initial or manual, reusing v2_report_run_trigger);
  • one row per user and run.
• Per-user columns of shared rows. reports.label, report_runs.trigger and budget_window_runs.trigger become nullable, with no default. reports.description and user_id already were. This lets the simulation service stop writing them. The report read and the run lists answer a missing label or trigger as null, and the /v2/reports response models allow it. A later revision (tracker task C1 · 41) drops them.
• Grants (privileges.py): the runtime role may read, insert and delete the new table's rows, and no other role has access. The simulation service needs nothing new: ON CONFLICT with a target needs SELECT and INSERT, which it already holds. No role may change a key, forced or a window run's policyengine_version.
• The one post-generation correction: the new table's trigger reuses the existing enum type (create_type=False).

It follows the shared-reports design decisions (tracker task C1 · 24):

• 1a, 1c, 4: the keys' contents, computed by the simulation service; key columns beside random row IDs.
• 1b: dynamics keyed by ID for now.
• 2: a failed run never counts.
• 3: a forced re-run is an extra, marked run.
• 6: the per-user columns go. This PR makes them nullable; C1 · 41 drops them.
• 7: a repeated request keeps the first requested_at, so there is no UPDATE grant.
• 8: household reports and budget windows are keyed now.
• 10: no user_simulation_run_associations.

Beyond the design's column list, the simulation-run index also covers year and country_package_version. That only allows rows the shorter index would refuse.

Deploy order and checks

• Order: apply this revision in staging and production before the simulation service's find-or-create change (C1 · 27, PolicyEngine/policyengine-sim-api). That change uses these indexes with INSERT ... ON CONFLICT DO NOTHING and stops writing the per-user columns.
  • Until it deploys, the simulation service's re-runs on unchanged versions answer 500, because the new indexes refuse the second run. This happens beside a succeeded run, and after a failed report run whose other side succeeded; household re-runs are included.
  • Creates are unaffected, and nothing deployed calls the re-run routes.
  • The Wales staging check's switch to forced re-runs has to come with or after that change.
• Before merging: the upgrade fails, changing nothing, if a database already has two runs that would break an index. Run the two read-only queries in the revision's section of docs/engineering/skills/alembic-migrations.md in staging and production first.
• The previous revision: it maps none of the new columns or table and writes none of these tables, so it keeps working while the simulation service still writes the per-user columns. Once that service writes nulls there, don't roll the Public API back below this revision: the previous revision takes label and trigger as required.
• The downgrade: it makes the per-user columns required again, so it fails while any report has no label or any run no trigger.
• The simulation service's schema check: its weekly comparison with this repository's tables fails after the merge until its table models follow; that is expected.

Changed existing tests

They stored two unfailed runs on the same versions, which the new indexes refuse. Each keeps its intent:

• a forced re-run (RunSettings.forced in tests/integration/stored_report_rows.py);
• each report side, budget-window year or withheld-codes run gets its own simulation;
• a run is finished instead of adding a second one.

test_alembic_v2, test_import_side_effects (49 tables), test_simulation_run_models and test_models now pin the new head, chain, table count, columns and indexes.

CI

• tests/integration/test_alembic_v2_shared_report_keys.py joins .github/scripts/test_alembic_v2_lifecycle.sh.
• tests/integration/test_v2_shared_report_rows.py joins the managed-roles step of v2-integration-check.yml.
• No environment variable is added. Both jobs already set V2_MIGRATION_DATABASE_URL and V2_ALEMBIC_DISPOSABLE_TEST, and the integration job DEPLOYMENT_ENVIRONMENT, as literal job env: values for their own PostgreSQL service container.
• The deploy's existing reconcile_v2_privileges.py apply/verify steps grant and check the new table. Those steps are unchanged.

Testing

• Database: a disposable PostgreSQL 17 database, built fresh with alembic -c alembic-v2.ini upgrade head, then scripts/reconcile_v2_privileges.py apply --create-missing-roles and verify.
• Drift: alembic -c alembic-v2.ini check found no new operations, and there is one head. Generating 94dc43079186 again on 1e1e3f079f2c gives the committed operations.
• Tests: every added or changed test, run individually by node ID (67 cases), passed, as did the tests the merge of master affects. Each new test was seen failing first.
• Rows: test_v2_shared_report_rows.py runs the simulation service's ON CONFLICT find-or-create, forced, failed and window runs, rows without per-user columns, and the runtime role's records, each as its managed role.
• Static checks:
  • ruff, ruff format, lint-imports, python -m scripts.rules validate, python scripts/run_quality_guards.py and mypy all passed.
  • Explicit mypy on the changed policyengine_api/data/v2/ files shows the 20 errors master has there, plus 2 of the same kind (SQLModel's Field overloads and sa_type=) on the new table's requested_at and trigger.
• Reviews: six adversarial review rounds.
  • Rounds 1 and 2 found existing unit tests that pinned the old revision chain, table count, columns and indexes.
  • Rounds 3 to 6 found no bugs. Their documentation and test fixes are included.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.45%. Comparing base (d7042b2) to head (63cce82).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4303      +/-   ##
==========================================
- Coverage   93.47%   93.45%   -0.03%     
==========================================
  Files         369      369              
  Lines       21455    21478      +23     
  Branches     3400     3401       +1     
==========================================
+ Hits        20056    20072      +16     
- Misses        816      823       +7     
  Partials      583      583              

Flag	Coverage Δ
integration	56.66% <100.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.