Skip to content

Record the UK area codes disclosure control withholds from a report run - #4305

Merged
anth-volk merged 5 commits into
masterfrom
v2plan/c1-22-local-area-schema
Nov 13, 2026
Merged

anth-volk merged 5 commits into
masterfrom
v2plan/c1-22-local-area-schema

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Fixes #4304

Summary

  • Adds the v2 table withheld_area_codes: one row per UK constituency or local authority that disclosure control withheld from a report run's local-area results. Columns: id, created_at, report_run_id (deleted with the report run), baseline_simulation_run_id and reform_simulation_run_id (RESTRICT, as the other comparison result tables have), area_type (a new native enum v2_uk_area_type: constituency, local_authority) and code (varchar(64)), unique on (report_run_id, area_type, code). A report run's ORM relationship deletes its rows with it.
  • Revision 1e1e3f079f2c, autogenerated (with --rev-id) on 0c136a278f29. Its one correction after generation is for reversibility: the downgrade also drops v2_uk_area_type, which Alembic's generated downgrade leaves behind, as 439303be14fe does for its enums. The upgrade only creates the type and the table, so it writes no data, and the previous revision of the Public API, which doesn't map the table, keeps working. The downgrade drops the table with its rows. V2_ALEMBIC_HEAD_REVISION moves to it, and the Alembic skill's list of revisions describes it.
  • Privileges: the Modal worker role gets SELECT, INSERT and the runtime role SELECT; the simulation service, Stage 12 and catalog writer roles get nothing. The deploy's existing steps (seed-v2-database.yml, from push.yml) upgrade the schema and then run reconcile_v2_privileges.py apply and verify, so no deploy step, secret or environment variable changes.
  • The table records which areas were withheld, not whether a run's list was written: until the simulation service's Modal workers write it, every run has no rows, so "no rows" means "none withheld" only for a succeeded run whose writer records the list. No route reads it yet.

The schema follows the local-area design's decision 7 (a new withheld_area_codes table, in the design's section 4.3 shape). The earlier plan's reports.parent_report_id is dropped: the design replaced it (decision 6) with ordinary national and local-area reports linked by a key, which a separate change adds.

Two other open PRs add v2 revisions on the same parent 0c136a278f29: #4303 and #4297. Whichever merges later moves its down_revision onto the new head and updates the tests that pin the chain (V2_ALEMBIC_HEAD_REVISION, the revision chain in test_alembic_v2, the table count in test_import_side_effects, and the post-baseline table set).

Testing

  • Tests first (2150a603f, failing), then the change. Unit: the model's columns, enum, unique key and relationship; the manifest's table sets; the revision chain and the revision's contents (generated, one enum drop); the CI lifecycle script listing the new integration module; the v2 table count.
  • Integration, against disposable PostgreSQL 17 (tests/integration/test_alembic_v2_withheld_area_codes.py, run by .github/scripts/test_alembic_v2_lifecycle.sh): the revision adds only the table and the type and changes no other table; the downgrade removes both and upgrading again works; a repeated (run, type, code) and an unknown area type are refused; a simulation run that withheld codes name can't be deleted; deleting the report run deletes its codes.
  • Integration, as the managed roles (test_v2_simulation_role_privileges): the Modal worker inserts and reads rows and can't update or delete them; the runtime role reads them and can't insert; the simulation service, Stage 12 and catalog writer roles can't read them.
  • Locally: the added and changed tests by node ID; alembic -c alembic-v2.ini check; a downgrade to base and an upgrade to head; reconcile_v2_privileges.py apply --create-missing-roles and verify; make quality (through the commit hook); python -m scripts.rules validate; the changelog fragment check.

🤖 Generated with Claude Code

anth-volk and others added 5 commits November 14, 2026 01:13
…ing)

The Modal worker records each UK area disclosure control withheld from a
report run in withheld_area_codes, keyed by the run and its two simulation
runs, with a v2_uk_area_type enum; the Public API reads them. The revision is
generated and its downgrade also drops the enum type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
withheld_area_codes holds one row per constituency or local authority a
local-area run withheld, keyed by the report run and its two simulation runs,
with the area type as the new v2_uk_area_type enum. The revision is generated,
with one documented correction: its downgrade also drops the enum type. The
Modal worker role reads and inserts the rows and the Public API's runtime role
reads them; the deploy's existing reconcile step grants both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check every managed role's access in the role test, say when an empty list
means no area was withheld, stop pinning the head in the revision's own test,
and list the table among the simulation side's tables in the simulation runs
revision's test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Say that an empty list means no area was withheld only for a succeeded run
whose writer records it, that the writer, not a constraint, keeps withheld
areas out of the area tables, and check that the catalog writer can't read
the rows either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3's quality findings: the writer, not the row, leaves a
withheld area out of the area tables, and the database can't tell a run
whose writer records the list from one whose writer doesn't. Rewrap the
skill entry's paragraph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.45%. Comparing base (f9c619d) to head (ceabfa1).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4305      +/-   ##
==========================================
- Coverage   93.47%   93.45%   -0.02%     
==========================================
  Files         369      369              
  Lines       21442    21455      +13     
  Branches     3400     3400              
==========================================
+ Hits        20043    20051       +8     
- Misses        816      822       +6     
+ Partials      583      582       -1     
Flag Coverage Δ
integration 56.62% <100.00%> (+0.02%) ⬆️
unit 91.01% <100.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anth-volk
anth-volk marked this pull request as ready for review November 13, 2026 22:22
@anth-volk
anth-volk merged commit ca4d396 into master Nov 13, 2026
23 checks passed
@anth-volk
anth-volk deleted the v2plan/c1-22-local-area-schema branch November 13, 2026 22:22
anth-volk added a commit to PolicyEngine/policyengine-sim-api that referenced this pull request Nov 13, 2026
ca4d396e merges PolicyEngine/policyengine-api#4305, whose revision
1e1e3f079f2c adds the withheld_area_codes table that Stage 12 writes a
local-area report run's withheld codes to. Since the previous pin it adds
no other migration and changes no table sim's persistence models describe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record the UK area codes disclosure control withholds from a report run

1 participant