Repository navigation
Add SQLAlchemy 2.1.x support and SQLAlchemy test suite integration - #206
bchoudhary6415 wants to merge 1 commit into
Conversation
Signed-off-by: Balram Choudhary <bchoudhary@rocketsoftware.com>
bimalkjha
left a comment
There was a problem hiding this comment.
Thanks for the work on this. I read the full diff and ran the PR in a temporary worktree against SQLAlchemy 2.0.46. I did not run it against a live Db2, so the dialect-behaviour comments come from reading the code. Requesting changes mainly for items 1, 2, 4 and 6.
Blocking
1. conftest.py overrides SQLAlchemy's pytest_collection_modifyitems
from sqlalchemy.testing.plugin.pytestplugin import *imports SQLAlchemy'spytest_collection_modifyitems.conftest.pythen defines a function with the same name later in the file, which replaces it.- SQLAlchemy's hook expands
__backend__classes per database and applies__only_on__,__requires__and the backend markers. - Confirmed by collecting
test/test_suite.py:- With the PR's hook, classes appear as
CTETest,ArgSignatureTestand so on, with no backend suffix, and pytest warns about an unknownpytest.mark.backend. - With the PR's hook renamed so SQLAlchemy's runs, classes appear as
BinaryTest_sqlite+pysqlite_3_50_4.
- With the PR's hook, classes appear as
- So the documented test names (
NumericTest_ibm_db_sa+ibm_db_sa_DB2/...) will not appear, and backend gating is silently off. - Suggested fix: call SQLAlchemy's hook first, then add the skip markers.
Alternatively, skip these cases through a requirement or exclusion instead of by node ID.
from sqlalchemy.testing.plugin import pytestplugin as _sa_plugin def pytest_collection_modifyitems(session, config, items): _sa_plugin.pytest_collection_modifyitems(session, config, items) ...
-p no:warningsinaddoptsinsetup.cfghides the symptom.
2. The default [db] URL is SQLite, so a plain pytest runs the compliance suite against SQLite
- Running
pytest test/test_suite.pywith no options fails on SQLite (DeprecatedCompoundSelectTest::test_limit_offset_selectable_in_unionsfailed first). That is misleading, because the PR body and docs presentpytest test/as the full run. - Suggested fix: make the Db2-requiring modules skip or error clearly unless the backend is Db2 (for example
__only_on__ = "db2"on those classes, or a collection guard in conftest), or have the default fail fast with a clear message.
Should fix
3. Packaging and docs are inconsistent
setup.pynow sayssqlalchemy>=1.4,<2.2and the README says 1.4–2.1.requirements-test.txtand the docs only test 2.0 and 2.1, so 1.4 is claimed but untested.- README says "Python 3.11 to 3.15", but
python_requiresis still>=3.9. Please align them. - Dropping 1.3 in
setup.pywhilebase.pyandibm_db.pystill carry 1.3 and 1.4 shims is a breaking change. It needs aCHANGES.mdentry and a version bump. Renaming thedevextra totestalso breakspip install ibm_db_sa[dev]. - I did not check whether
_de_cloneexists in 2.1. Thetry/except Exceptionfallback silently turns it into a no-op, which would hide a break rather than surface it.
4. base.py: the OFFSET rewrite is a regression
- The old regex was anchored to the end (
\s+OFFSET ...\s*$). The new loop takes the last\bOFFSET\banywhere in the SQL and cuts everything after it. - A statement with no trailing OFFSET but one in a subquery or CTE (or in a literal or identifier) would be truncated and corrupted.
- Please keep an end anchor, or check that the captured tail is a single offset token or bind.
FETCH FIRST/LIMITtext after the OFFSET would also end up inoffset_text.
5. base.py: CTAS and type DDL changes
visit_create_table_asignoreselement.temporaryandelement.if_not_exists.visit_unicode_textnow emitsDBCLOBinstead ofLONG VARGRAPHIC. That is a DDL change for every user and is not mentioned in the PR description.visit_enumemitsVARCHAR(n CODEUNITS32), which is only valid on newer LUW and on z/OS. Please document the minimum server versions, or gate it.
6. base.py: visit_BOOLEAN and visit_enum read DB2Dialect.serverType (class-level global state)
- It is set by whichever dialect connected last. Two engines in one process (for example LUW and AS400) would render the wrong DDL.
- Prefer
self.dialect.dbms_name, asvisit_empty_set_op_expralready does.
7. reflection.py: caching and normalization
@reflection.cacheonhas_tablemeans an Inspector reused acrossCREATEandDROPreturns stale results untilclear_cache(). The new_ensure_table_existsalso adds an extra query to every reflection call.normalize_nameno longer calls_requires_quotes. Uppercase names that need quoting now become lowercase, and mixed-case names becomequoted_name(quote=True). This is a behaviour change for existing users and needs a regression test.- A genuine unique index created with
EXCLUDE NULL KEYSis now indistinguishable from a UNIQUE constraint. That is acceptable, but please document it.
8. ibm_db.py: _homogenize_numeric_columns
- It silently converts int, float and Decimal values within a column in
executemany. - On SQLAlchemy 1.4,
float(Decimal)andfloat(int)lose precision for large values. - It runs on every
executemanyand scans every row and column. It should be scoped to Numeric/Float columns. - The
supports_unicode_bindscomment is cut off mid-sentence ("...pre-encodes to UTF-8 bytes and").
9. visit_bindparam and render_literal_value
visit_bindparamwrapsBigIntegerbinds inCAST(... AS BIGINT)in every clause. This changes the SQL for every statement that uses a BigInteger bind, and theexcept Exceptionswallows errors. Please restrict it to the places that need it and narrow the exception.render_literal_valuereturnsUX'...'for any non-ASCII string. It is untested, and unverified for z/OS and AS400.
Tests and cleanup
test/test_dialect.pyhas 7 weak tests.test_limit_rendersandtest_offset_uses_rownumberonly check for substrings.- None of the new compiler logic is covered: the truediv/floordiv casts, the bind CAST, the empty-IN rewrite, the CTAS rendering, the Enum and Boolean AS400 branches,
render_literal_value, the OFFSET capture, and thenormalize_nameanddenormalize_namechanges. - These are all compile-only and can use
AssertsCompiledSQLwithout a database. test_dialect.pycallsregistry.registerat import time, which duplicatesconftest.pyand is not needed.- Nits:
quote_identifieris now indented 10 spaces in a 3-space file, and its docstring was removed.- Some comments are one-off test notes (for example
test_get_view_definition_does_not_exist) and will rot. requirements-test.txthas a stray trailing blank line.README.unittests.mdshould mention that a plainpytestruns against SQLite (see item 2).
Summary
This PR adds support for SQLAlchemy 2.1.x to ibm_db_sa. It also lets
SQLAlchemy's pytest-based test suite for third-party dialects run against
ibm_db_sa, so the dialect can be tested with different SQLAlchemy versions.
Changes
conftest.pyregisters the Db2 dialect and loads SQLAlchemy's pytest plugin.setup.cfgholds the test settings and database URLs.ibm_db_sa/requirements.pylists the features Db2 supports.setup.pyis updated, and thenoseextra is replaced with apytestextra.requirements-test.txt.MANIFEST.innow includes the test files.README.mdand a full guide inREADME.unittests.md.How to test