Skip to content

Core: Compare upsert rows column by column instead of cell by cell - #4334

Open
Anurag-S1ngh wants to merge 1 commit into
apache:mainfrom
Anurag-S1ngh:perf/3860-vectorize-upsert-row-comparison
Open

Anurag-S1ngh wants to merge 1 commit into
apache:mainfrom
Anurag-S1ngh:perf/3860-vectorize-upsert-row-comparison

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown

Rationale for this change

get_rows_to_update compared every matched row one cell at a time, slicing both tables and calling as_py() for each column. Cost grew with rows times columns: on a table with 200 columns and 20k matched rows the comparison alone took around 20 seconds, before anything was written.

The matched rows are now gathered once and compared column by column with PyArrow kernels. Two nulls count as equal, a null against a value counts as a change, and every row is still returned only once. Columns whose types have no comparison kernel, structs and lists today, fall back to the previous value-by-value comparison, so nested types behave as before.

The reproduction from the issue now takes about 0.1 seconds instead of about 20.

Are these changes tested?

Yes. New tests in tests/table/test_upsert.py cover get_rows_to_update directly:

  • only the changed rows are returned
  • a row is returned once even when several columns changed
  • two nulls compare equal while a null against a value is a change
  • struct columns are still compared

Full unit suite: 4570 passed, 5 skipped. prek run -a passes.

Are there any user-facing changes?

Yes, only in speed: upsert no longer slows down sharply with the number of columns. The returned rows are unchanged, except that two NaN values in the same column now compare equal, which matches Iceberg's comparator semantics.

Closes #3860

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upsert gets slow on tables with many columns

1 participant