Skip to content

Add SQLAlchemy 2.1.x support and SQLAlchemy test suite integration - #206

Open
bchoudhary6415 wants to merge 1 commit into
ibmdb:masterfrom
bchoudhary6415:ibmdbsa_test
Open

bchoudhary6415 wants to merge 1 commit into
ibmdb:masterfrom
bchoudhary6415:ibmdbsa_test

Conversation

@bchoudhary6415

Copy link
Copy Markdown
Collaborator

Summary

This PR adds support for SQLAlchemy 2.1.x to ibm_db_sa. It also lets
SQLAlchemy's pytest-based test suite for third-party dialects run against
ibm_db_sa, so the dialect can be tested with different SQLAlchemy versions.

Changes

  • SQLAlchemy 2.1.x support: ibm_db_sa now supports SQLAlchemy 1.4.x through 2.1.x.
  • Testing:
    • conftest.py registers the Db2 dialect and loads SQLAlchemy's pytest plugin.
    • setup.cfg holds the test settings and database URLs.
    • ibm_db_sa/requirements.py lists the features Db2 supports.
    • The suite also includes Db2-specific tests.
    • To test another SQLAlchemy version, install it and rerun the tests; no test changes are needed.
  • Packaging:
    • The SQLAlchemy requirement in setup.py is updated, and the nose extra is replaced with a pytest extra.
    • New requirements-test.txt.
    • MANIFEST.in now includes the test files.
  • Docs: New Testing section in README.md and a full guide in README.unittests.md.

How to test

pip install -e .
pip install -r requirements-test.txt
pytest test/test_dialect.py                       # no database needed
pytest --dburi "ibm_db_sa://user:pass@host:port/DBNAME" test/   # against Db2

Signed-off-by: Balram Choudhary <bchoudhary@rocketsoftware.com>
@bchoudhary6415 bchoudhary6415 self-assigned this Nov 17, 2026

@bimalkjha bimalkjha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work on this. I read the full diff and ran the PR in a temporary worktree against SQLAlchemy 2.0.46. I did not run it against a live Db2, so the dialect-behaviour comments come from reading the code. Requesting changes mainly for items 1, 2, 4 and 6.

Blocking

1. conftest.py overrides SQLAlchemy's pytest_collection_modifyitems

  • from sqlalchemy.testing.plugin.pytestplugin import * imports SQLAlchemy's pytest_collection_modifyitems. conftest.py then defines a function with the same name later in the file, which replaces it.
  • SQLAlchemy's hook expands __backend__ classes per database and applies __only_on__, __requires__ and the backend markers.
  • Confirmed by collecting test/test_suite.py:
    • With the PR's hook, classes appear as CTETest, ArgSignatureTest and so on, with no backend suffix, and pytest warns about an unknown pytest.mark.backend.
    • With the PR's hook renamed so SQLAlchemy's runs, classes appear as BinaryTest_sqlite+pysqlite_3_50_4.
  • So the documented test names (NumericTest_ibm_db_sa+ibm_db_sa_DB2/...) will not appear, and backend gating is silently off.
  • Suggested fix: call SQLAlchemy's hook first, then add the skip markers.
    from sqlalchemy.testing.plugin import pytestplugin as _sa_plugin
    
    def pytest_collection_modifyitems(session, config, items):
        _sa_plugin.pytest_collection_modifyitems(session, config, items)
        ...
    Alternatively, skip these cases through a requirement or exclusion instead of by node ID.
  • -p no:warnings in addopts in setup.cfg hides the symptom.

2. The default [db] URL is SQLite, so a plain pytest runs the compliance suite against SQLite

  • Running pytest test/test_suite.py with no options fails on SQLite (DeprecatedCompoundSelectTest::test_limit_offset_selectable_in_unions failed first). That is misleading, because the PR body and docs present pytest test/ as the full run.
  • Suggested fix: make the Db2-requiring modules skip or error clearly unless the backend is Db2 (for example __only_on__ = "db2" on those classes, or a collection guard in conftest), or have the default fail fast with a clear message.

Should fix

3. Packaging and docs are inconsistent

  • setup.py now says sqlalchemy>=1.4,<2.2 and the README says 1.4–2.1. requirements-test.txt and the docs only test 2.0 and 2.1, so 1.4 is claimed but untested.
  • README says "Python 3.11 to 3.15", but python_requires is still >=3.9. Please align them.
  • Dropping 1.3 in setup.py while base.py and ibm_db.py still carry 1.3 and 1.4 shims is a breaking change. It needs a CHANGES.md entry and a version bump. Renaming the dev extra to test also breaks pip install ibm_db_sa[dev].
  • I did not check whether _de_clone exists in 2.1. The try/except Exception fallback silently turns it into a no-op, which would hide a break rather than surface it.

4. base.py: the OFFSET rewrite is a regression

  • The old regex was anchored to the end (\s+OFFSET ...\s*$). The new loop takes the last \bOFFSET\b anywhere in the SQL and cuts everything after it.
  • A statement with no trailing OFFSET but one in a subquery or CTE (or in a literal or identifier) would be truncated and corrupted.
  • Please keep an end anchor, or check that the captured tail is a single offset token or bind.
  • FETCH FIRST/LIMIT text after the OFFSET would also end up in offset_text.

5. base.py: CTAS and type DDL changes

  • visit_create_table_as ignores element.temporary and element.if_not_exists.
  • visit_unicode_text now emits DBCLOB instead of LONG VARGRAPHIC. That is a DDL change for every user and is not mentioned in the PR description.
  • visit_enum emits VARCHAR(n CODEUNITS32), which is only valid on newer LUW and on z/OS. Please document the minimum server versions, or gate it.

6. base.py: visit_BOOLEAN and visit_enum read DB2Dialect.serverType (class-level global state)

  • It is set by whichever dialect connected last. Two engines in one process (for example LUW and AS400) would render the wrong DDL.
  • Prefer self.dialect.dbms_name, as visit_empty_set_op_expr already does.

7. reflection.py: caching and normalization

  • @reflection.cache on has_table means an Inspector reused across CREATE and DROP returns stale results until clear_cache(). The new _ensure_table_exists also adds an extra query to every reflection call.
  • normalize_name no longer calls _requires_quotes. Uppercase names that need quoting now become lowercase, and mixed-case names become quoted_name(quote=True). This is a behaviour change for existing users and needs a regression test.
  • A genuine unique index created with EXCLUDE NULL KEYS is now indistinguishable from a UNIQUE constraint. That is acceptable, but please document it.

8. ibm_db.py: _homogenize_numeric_columns

  • It silently converts int, float and Decimal values within a column in executemany.
  • On SQLAlchemy 1.4, float(Decimal) and float(int) lose precision for large values.
  • It runs on every executemany and scans every row and column. It should be scoped to Numeric/Float columns.
  • The supports_unicode_binds comment is cut off mid-sentence ("...pre-encodes to UTF-8 bytes and").

9. visit_bindparam and render_literal_value

  • visit_bindparam wraps BigInteger binds in CAST(... AS BIGINT) in every clause. This changes the SQL for every statement that uses a BigInteger bind, and the except Exception swallows errors. Please restrict it to the places that need it and narrow the exception.
  • render_literal_value returns UX'...' for any non-ASCII string. It is untested, and unverified for z/OS and AS400.

Tests and cleanup

  • test/test_dialect.py has 7 weak tests. test_limit_renders and test_offset_uses_rownumber only check for substrings.
  • None of the new compiler logic is covered: the truediv/floordiv casts, the bind CAST, the empty-IN rewrite, the CTAS rendering, the Enum and Boolean AS400 branches, render_literal_value, the OFFSET capture, and the normalize_name and denormalize_name changes.
  • These are all compile-only and can use AssertsCompiledSQL without a database.
  • test_dialect.py calls registry.register at import time, which duplicates conftest.py and is not needed.
  • Nits:
    • quote_identifier is now indented 10 spaces in a 3-space file, and its docstring was removed.
    • Some comments are one-off test notes (for example test_get_view_definition_does_not_exist) and will rot.
    • requirements-test.txt has a stray trailing blank line.
    • README.unittests.md should mention that a plain pytest runs against SQLite (see item 2).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants