Skip to content

Take data_source on /v2/reports, answer each report's dataset_independent_key, and find reports by it - #4310

Draft
anth-volk wants to merge 14 commits into
v2plan/c1-25-shared-report-keysfrom
v2plan/c1-30-data-source-and-keys
Draft

anth-volk wants to merge 14 commits into
v2plan/c1-25-shared-report-keysfrom
v2plan/c1-30-data-source-and-keys

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4309

Summary

Stacks on #4303 (v2plan/c1-25-shared-report-keys, which adds reports.dataset_independent_key and ix_reports_dataset_independent_key), and includes #4307's three commits until #4307 merges: #4307 stops the Public API sending dataset_name to the simulation service and makes POST /v2/reports and POST /v2/simulations refuse it, so this PR adds no dataset_name change of its own. Retarget to master once #4303 merges.

  • data_source on POST /v2/reports. An economy report takes data_source: "local_area_dataset", left out unless given (as spm): an explicit null or another value is refused with 422, and budget windows and household reports refuse the field as unknown. It is sent to the simulation service only when given. The Public API checks only the value; the simulation service checks the country (uk), the region (uk and the four country/* nations) and whether the release gives local areas a dataset of their own. Its 422s answer 422 with its code and reasons, and its 400 local_areas_unsupported and local_area_dataset_mismatch (its own state) answer 503, through the existing client mapping. The value is defined once (ReportDataSource, LOCAL_AREA_DATASET).
  • dataset_independent_key on every report read. Every report item of every kind (detail, list, search, and the create and re-run answers) answers the stored reports.dataset_independent_key, null for a report stored before the keys. The Public API never computes a key.
  • The search: GET /v2/reports?country_id=…&dataset_independent_key=…. The list route takes either report_id (repeated, at most 25, as before) or one dataset_independent_key (64 lowercase hexadecimal digits), never both or neither (422). It answers the country's reports with that key as summary items, newest created first (ties to the larger ID), at most the 25 created last, through ix_reports_dataset_independent_key. No credential, as every report read; the runtime role's existing SELECT on reports.
  • A local-area report's read. A UK-wide or nation economy report whose dataset is the default dataset of both a constituency region and a local-authority region of its catalog version is a local-area report, as the simulation service decides which dataset local areas use; nothing stores the request's data_source. Its read says data_source: "local_area_dataset" (null for every other economy report), and its result adds constituency_withheld_codes and local_authority_withheld_codes from withheld_area_codes (Record the UK area codes disclosure control withholds from a report run #4305), sorted, [] when none were withheld; an area level whose areas were all withheld reads []. Every other result leaves the two lists out, as the simulation service's SingleYearMacroOutput does (Calculate UK-wide and nation reports on the local-area dataset, with their areas and withheld codes policyengine-sim-api#1068). The withheld codes are read only for a local-area report.

What must merge and deploy first:

Migration: no route group moves; /v2/reports stays on FastAPI. User-visible contract changes: the new request field, the new query parameter (and report_id becoming optional when the key is given), and the new read fields. The generated migration contract documents are unchanged by the export.

Inline suppression added: # noqa: PLR0913 on store_report in tests/integration/stored_report_rows.py (the connection and catalog, then each column a test varies): it gains two keyword arguments for the key and the creation time, and grouping its existing arguments would change every caller.

Testing

Each change was built test first (the failing tests are their own commits). Run by node ID only:

  • unit: tests/unit/v2/test_report_dataset_independent_keys.py (SQLite: every kind of read answers the key; the search's country, order, ties, limit and empty answer), tests/unit/v2/test_local_area_report_reads.py (SQLite: which reports read as local-area reports), tests/unit/v2/test_economy_result_rebuild.py (withheld codes rebuilt sorted; an empty level; other results leave the lists out), tests/unit/v2/test_report_read_routes.py (the key, the search route, its refused queries, OpenAPI), tests/unit/v2/test_report_create_routes.py and tests/unit/libs/test_simulation_reports.py (data_source passed on and refused; the simulation service's refusals, as stand-ins shaped as its C1 · 29a answers);
  • integration, PostgreSQL 17 as the runtime role (tests/integration/test_v2_report_reads.py): the search; a local-area report's areas and withheld codes; an empty level; UK-wide and nation reports on the national dataset;
  • lint-imports, python -m scripts.rules validate, python scripts/run_quality_guards.py, ruff, ruff format, mypy, and python scripts/export_migration_contracts.py (no change). No revision or grant change.

Two read-only review rounds: round 1 found no bugs (documentation, one shared value, test setup and coverage fixed); round 2, on those fixes, found no bugs (test and documentation fixes only).

🤖 Generated with Claude Code

anth-volk and others added 14 commits November 14, 2026 02:19
The Public API's report, budget-window and simulation requests to the
simulation service send no dataset_name, and POST /v2/reports and
POST /v2/simulations refuse the field, given or null.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The simulation service is dropping dataset_name from its report,
budget-window and simulation requests, and it refuses unknown fields, so
the Public API stops sending the field first. POST /v2/reports and
POST /v2/simulations stop taking it too: their request models no longer
inherit it, so a body naming it, even as null, is refused with 422. Reads
still name the dataset a report or simulation used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1's documentation findings: a creation no longer names a
dataset, so the client's module text, the create routes' 503 descriptions,
the caller-row refusal codes' comment and a test's comment say the
caller names a region or dynamic, and the dataset is the region's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y it (failing)

Every report read answers reports.dataset_independent_key, and
GET /v2/reports takes dataset_independent_key in place of report_id to
find the country's reports that share it, newest first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every report item, of every kind, answers reports.dataset_independent_key
as the simulation service stored it, null for a report stored before the
keys. GET /v2/reports takes dataset_independent_key in place of
report_id and lists the country's reports that share it, newest created
first, at most the 25 created last, through
ix_reports_dataset_independent_key. The query takes one of the two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e (failing)

POST /v2/reports takes data_source: "local_area_dataset" on an economy
report and sends it on only when given; null, another value and the
field on a budget window or household report are refused before the
simulation service is asked. The simulation service checks the region,
the country and the release: its 422 answers 422, and its 400 for its
own local-area dataset state 503.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EconomyReportFields takes data_source: "local_area_dataset", left out
unless given, as spm is; an explicit null or another value is refused
with 422, and budget windows and household reports refuse the field as
unknown. The report request sends it only when given. The simulation
service checks the region, the country and the release, and its
refusals map as before: a 422 answers 422, and a 400 about its own
local-area dataset 503.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UK-wide or nation report whose dataset is the one the catalog version
gives both the constituency and local-authority regions is a local-area
report: its read says data_source local_area_dataset, and its result
lists each level's withheld area codes, sorted, with a level whose areas
were all withheld read as empty. Every other result leaves the two lists
out, as the simulation service's does. The result fixture stores
withheld_area_codes as PolicyEngine/policyengine-sim-api#1068's writer
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UK-wide or nation report whose dataset is the default dataset of both
the constituency and the local-authority regions of its catalog version
is a local-area report, as the simulation service decides which dataset
local areas use. Its read says data_source local_area_dataset, its
result lists each level's withheld area codes from withheld_area_codes,
sorted, and a level whose areas were all withheld reads as empty. Every
other report reads data_source null, and its result leaves the two lists
out, as the simulation service's SingleYearMacroOutput does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two report items' docstrings keep both sides: the null label and the
dataset-independent key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The request's ReportDataSource and its LOCAL_AREA_DATASET value now name
the read's data_source too. The list result's docstring gives the
search's order, and the read's data_source says it comes from the
report's region and dataset. The stand-in refusals match the simulation
service's answers, the two SQLite read tests share their setup
(tests.unit.v2.sqlite_report_rows, which drops one suppression), and
PostgreSQL now reads a UK-wide report on the national dataset and a
local-area report with an empty area level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The simulation service's two data_source refusals now answer the UK
requests that draw them: a UK-wide report on a release without a
local-area dataset, and a constituency's own report. The value's comment
says a local area's own report never reads it, and the economy item's
docstring reads in order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.60656% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.44%. Comparing base (63cce82) to head (8f9771d).

Files with missing lines Patch % Lines
policyengine_api/libs/simulation_reports.py 0.00% 4 Missing ⚠️
...engine_api/services/v2/simulation_service/types.py 63.63% 4 Missing ⚠️
...yengine_api/services/v2/stored_reports/services.py 77.77% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@                         Coverage Diff                         @@
##           v2plan/c1-25-shared-report-keys    #4310      +/-   ##
===================================================================
- Coverage                            93.45%   93.44%   -0.01%     
===================================================================
  Files                                  369      369              
  Lines                                21478    21531      +53     
  Branches                              3401     3412      +11     
===================================================================
+ Hits                                 20072    20120      +48     
- Misses                                 823      832       +9     
+ Partials                               583      579       -4     
Flag Coverage Δ
integration 56.73% <83.60%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant