SQLAlchemy 1.4 support + Move CI to GitHub workflows #506
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
follow up to kvesteri#372
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
- postgres: return for the first positive test - use immutable for default argument
…s/sqlalchemy-utils into topic/372-followup
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
- use a connection (which is closed automatically) for data base existence check - explicitely use Null connection pool already with the 1st change disposal of the engine (which closes all open connections) is not necessary anymore. with the second change we are completely sure.
Import _ColumnEntity from sqlalchemy.orm.context if importing from .query fails. And while checking if database_exists, use url.set() as URL object is now immutable.
…*once* the version of sqlalchemy in use and deciding *once* how to change the database address. Using a wrapping function ('set_database_from_url') to simplify the code.
to set the transaction isolation level to autocommit for PostgreSQL.
Consolidate autocommit across drivers that support it, xref. kvesteri#494 and fix kvesteri#486 .
to use `with engine.connect()` context manager (commit 4f52578 )
when testing. xref. kvesteri#500 Fix the following exception: ``` _____________________________________________________________________________ TestDatabasePostgresPg8000.test_create_and_drop _____________________________________________________________________________ self = <tests.functions.test_database.TestDatabasePostgresPg8000 object at 0x7f3b3da76490>, dsn = 'postgresql+pg8000://postgres:postgres@localhost/db_to_test_create_and_drop_via_pg8000_driver' def test_create_and_drop(self, dsn): > assert not database_exists(dsn) tests/functions/test_database.py:15: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sqlalchemy_utils/functions/database.py:488: in database_exists return bool(_get_scalar_result(engine, text)) sqlalchemy_utils/functions/database.py:443: in _get_scalar_result with engine.connect() as conn: .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2263: in connect return self._connection_cls(self, **kwargs) .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:104: in __init__ else engine.raw_connection() .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2369: in raw_connection return self._wrap_pool_connect( .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2336: in _wrap_pool_connect return fn() .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:304: in unique_connection return _ConnectionFairy._checkout(self) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:778: in _checkout fairy = _ConnectionRecord.checkout(pool) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:495: in checkout rec = pool._do_get() .venv/lib/python3.8/site-packages/sqlalchemy/pool/impl.py:241: in _do_get return self._create_connection() .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:309: in _create_connection return _ConnectionRecord(self) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:440: in __init__ self.__connect(first_connect_check=True) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:664: in __connect pool.dispatch.first_connect.for_modify( .venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:314: in exec_once_unless_exception self._exec_once_impl(True, *args, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:285: in _exec_once_impl self(*args, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:322: in __call__ fn(*args, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py:1406: in go return once_fn(*arg, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py:199: in first_connect dialect.initialize(c) .venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/pg8000.py:215: in initialize super(PGDialect_pg8000, self).initialize(connection) .venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2624: in initialize super(PGDialect, self).initialize(connection) .venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:311: in initialize self.server_version_info = self._get_server_version_info( .venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2869: in _get_server_version_info v = connection.execute("select version()").scalar() .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1003: in execute return self._execute_text(object_, multiparams, params) .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1172: in _execute_text ret = self._execute_context( .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1316: in _execute_context self._handle_dbapi_exception( .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1514: in _handle_dbapi_exception util.raise_(exc_info[1], with_traceback=exc_info[2]) .venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py:182: in raise_ raise exception .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1294: in _execute_context result = context._setup_crud_result_proxy() .venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1258: in _setup_crud_result_proxy result = self.get_result_proxy() .venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1233: in get_result_proxy return result.ResultProxy(self) .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:775: in __init__ self._init_metadata() .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:807: in _init_metadata self._metadata = ResultMetaData(self, cursor_description) .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:290: in __init__ raw = self._merge_cursor_description( .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in _merge_cursor_description return [ .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in <listcomp> return [ .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:616: in _merge_cols_by_none for ( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <sqlalchemy.engine.result.ResultMetaData object at 0x7f3b3daa4a90>, context = <sqlalchemy.dialects.postgresql.pg8000.PGExecutionContext_pg8000 object at 0x7f3b3dda7100> cursor_description = [('version', 25, None, None, None, None, ...)] def _colnames_from_description(self, context, cursor_description): """Extract column names and data types from a cursor.description. Applies unicode decoding, column translation, "normalization", and case sensitivity rules to the names based on the dialect. """ dialect = context.dialect case_sensitive = dialect.case_sensitive translate_colname = context._translate_colname description_decoder = ( dialect._description_decoder if dialect.description_encoding else None ) normalize_name = ( dialect.normalize_name if dialect.requires_name_normalize else None ) untranslated = None self.keys = [] for idx, rec in enumerate(cursor_description): colname = rec[0] coltype = rec[1] if description_decoder: > colname = description_decoder(colname) E TypeError: expected bytes, str found .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:545: TypeError ```
|
TODO:
|
nsoranzo
commented
Apr 9, 2021
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
|
@nsoranzo the test setup is failing in CI |
|
Yeah, trying to add an health check for the SQL Server to verify it successfully started in the docker container before the tests start (I've seen some random failures when tests try to login to MSSQL). I can move the commit to a separate branch and keep testing there. |
nsoranzo
force-pushed
the
sqlalchemy14
branch
2 times, most recently
from
202a644
to
f0020cf
Compare
Remove `!` from password so `SA_PASSWORD` doesn't need extra quotes.
I've added the MS SQL health check, but that did not solve the issue. I still see around 1 out of 5 builds randomly fail tests/functions/test_database.py::TestDatabaseMssql::test_create_and_drop with (e.g. https://github.com/nsoranzo/sqlalchemy-utils/runs/2307689171?check_suite_focus=true ) |
@kvesteri SA 1.4.7 is out and this is basically green apart from the random mssql issue mentioned above. |
Change EncryptedType tests to use StringEncryptedType
StdCarrot
pushed a commit
to StdCarrot/sqlalchemy-utils
that referenced
this pull request
Apr 14, 2022
* fix database existence check follow up to kvesteri#372 * always dispose engine after db existence check * add docs for postgres_db parameter * fix dialect_name Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * optimize execution order Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * database_exists fix return - postgres: return for the first positive test - use immutable for default argument * use None as default Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * break if successful Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * dispose only for exception * use Null connection pool and close connections - use a connection (which is closed automatically) for data base existence check - explicitely use Null connection pool already with the 1st change disposal of the engine (which closes all open connections) is not necessary anymore. with the second change we are completely sure. * fix isort call in tox.ini and import order * rename parameter to databases * move functions to module level * Add support for SQLAlchemy 1.4 Import _ColumnEntity from sqlalchemy.orm.context if importing from .query fails. And while checking if database_exists, use url.set() as URL object is now immutable. * Fixed errors related to URL.database not being directly settable. * Removed the try..except constructs to set the database. Now checking *once* the version of sqlalchemy in use and deciding *once* how to change the database address. Using a wrapping function ('set_database_from_url') to simplify the code. * Cleanups * Use `execution_options()` method of `Connection` to set the transaction isolation level to autocommit for PostgreSQL. * Lint fixes * Move CI lint job to GitHub workflows * Testing: drop Python 3.5 and Python 3.9 * Use `with engine.connect()` context manager Consolidate autocommit across drivers that support it, xref. kvesteri#494 and fix kvesteri#486 . * Move CI tests from TravisCI to GitHub workflows * Add SQLAlchemy 1.3 on Python 3.6 to the test matrix * Update tests for changes in `create_database` to use `with engine.connect()` context manager (commit 4f52578 ) * Install the version of pg8000 recommended by SQLAlchemy 1.3 when testing. xref. kvesteri#500 Fix the following exception: ``` _____________________________________________________________________________ TestDatabasePostgresPg8000.test_create_and_drop _____________________________________________________________________________ self = <tests.functions.test_database.TestDatabasePostgresPg8000 object at 0x7f3b3da76490>, dsn = 'postgresql+pg8000://postgres:postgres@localhost/db_to_test_create_and_drop_via_pg8000_driver' def test_create_and_drop(self, dsn): > assert not database_exists(dsn) tests/functions/test_database.py:15: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sqlalchemy_utils/functions/database.py:488: in database_exists return bool(_get_scalar_result(engine, text)) sqlalchemy_utils/functions/database.py:443: in _get_scalar_result with engine.connect() as conn: .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2263: in connect return self._connection_cls(self, **kwargs) .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:104: in __init__ else engine.raw_connection() .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2369: in raw_connection return self._wrap_pool_connect( .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2336: in _wrap_pool_connect return fn() .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:304: in unique_connection return _ConnectionFairy._checkout(self) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:778: in _checkout fairy = _ConnectionRecord.checkout(pool) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:495: in checkout rec = pool._do_get() .venv/lib/python3.8/site-packages/sqlalchemy/pool/impl.py:241: in _do_get return self._create_connection() .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:309: in _create_connection return _ConnectionRecord(self) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:440: in __init__ self.__connect(first_connect_check=True) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:664: in __connect pool.dispatch.first_connect.for_modify( .venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:314: in exec_once_unless_exception self._exec_once_impl(True, *args, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:285: in _exec_once_impl self(*args, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:322: in __call__ fn(*args, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py:1406: in go return once_fn(*arg, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py:199: in first_connect dialect.initialize(c) .venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/pg8000.py:215: in initialize super(PGDialect_pg8000, self).initialize(connection) .venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2624: in initialize super(PGDialect, self).initialize(connection) .venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:311: in initialize self.server_version_info = self._get_server_version_info( .venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2869: in _get_server_version_info v = connection.execute("select version()").scalar() .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1003: in execute return self._execute_text(object_, multiparams, params) .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1172: in _execute_text ret = self._execute_context( .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1316: in _execute_context self._handle_dbapi_exception( .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1514: in _handle_dbapi_exception util.raise_(exc_info[1], with_traceback=exc_info[2]) .venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py:182: in raise_ raise exception .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1294: in _execute_context result = context._setup_crud_result_proxy() .venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1258: in _setup_crud_result_proxy result = self.get_result_proxy() .venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1233: in get_result_proxy return result.ResultProxy(self) .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:775: in __init__ self._init_metadata() .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:807: in _init_metadata self._metadata = ResultMetaData(self, cursor_description) .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:290: in __init__ raw = self._merge_cursor_description( .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in _merge_cursor_description return [ .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in <listcomp> return [ .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:616: in _merge_cols_by_none for ( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <sqlalchemy.engine.result.ResultMetaData object at 0x7f3b3daa4a90>, context = <sqlalchemy.dialects.postgresql.pg8000.PGExecutionContext_pg8000 object at 0x7f3b3dda7100> cursor_description = [('version', 25, None, None, None, None, ...)] def _colnames_from_description(self, context, cursor_description): """Extract column names and data types from a cursor.description. Applies unicode decoding, column translation, "normalization", and case sensitivity rules to the names based on the dialect. """ dialect = context.dialect case_sensitive = dialect.case_sensitive translate_colname = context._translate_colname description_decoder = ( dialect._description_decoder if dialect.description_encoding else None ) normalize_name = ( dialect.normalize_name if dialect.requires_name_normalize else None ) untranslated = None self.keys = [] for idx, rec in enumerate(cursor_description): colname = rec[0] coltype = rec[1] if description_decoder: > colname = description_decoder(colname) E TypeError: expected bytes, str found .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:545: TypeError ``` * Fix another import for SQLAlchemy 1.4 * Fix test failing on SQLAlchemy 1.4 * Fix `AttributeError: 'Query' object has no attribute '_entities'` * Fix `AttributeError: type object 'User' has no attribute '_decl_class_registry'` and similar messages for other objects. * Extend linting to all Python files * Fix `AttributeError: 'Query' object has no attribute '_mapper_zero'` * Fix `AttributeError: module 'sqlalchemy.orm.mapper' has no attribute '_mapper_registry'` * Fix `make_order_by_deterministic` for SQLAlchemy 1.4 Fix errors: ``` def make_order_by_deterministic(query): """ order_by_func = sa.asc > if not query._order_by: E AttributeError: 'Query' object has no attribute '_order_by' ``` ``` tests/functions/test_make_order_by_deterministic.py:74: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ query = <sqlalchemy.orm.query.Query object at 0x7f03d5af64f0> def make_order_by_deterministic(query): else: order_by_func = sa.asc > column = order_by.get_children()[0] E TypeError: 'itertools.chain' object is not subscriptable ``` * Add message to assert * Debug test failures under SQLAlchemy 1.4 * Don't mask potential `AttributeError` exceptions in `sa.engine.URL.create()` * Fix wrong `dialect_name` check Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * Remove `u` unnecessary on Python 3 * Drop unnecessary `alias()` call This returns a `Subquery` object which is not executable in SQLAlchemy 1.4: ``` tests/relationships/test_select_correlated_expression.py:399: assert session.execute(aggregate) .venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py:1587: in execute statement = coercions.expect(roles.StatementRole, statement) .venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:200: in expect return impl._implicit_coercions( .venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:836: in _implicit_coercions return super(StatementImpl, self)._implicit_coercions( .venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:242: in _implicit_coercions self._raise_for_expected(element, argname, resolved) .venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:270: in _raise_for_expected util.raise_(exc.ArgumentError(msg, code=code), replace_context=err) E sqlalchemy.exc.ArgumentError: Executable SQL or text() construct expected, got <sqlalchemy.sql.selectable.Subquery at 0x7fb34b5e1880; test>. ``` * fix get_columns tests * fix instant defaults listener * remove sort_query and get_query_entities The sort_query and get_query_entities functions never worked fully as intended and contained lots of quirky edge cases: 1. sort_query function was dangerous in a sense that it could be used for really inefficient queries (sorting by non-indexed column). 2. The entity string label introspection in both functions relied on SQLAlchemy internals which were drastically changed in SA 1.4. Relying on those was never a good idea in the first place. * Remove also `get_query_entity_by_alias()` which depends on `get_query_entities()` which was removed in commit 460e1da . * Fix linting errors * fix scalar_subquery warnings * Update sqlalchemy_utils/compat.py Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * fix selectable columns warning * Add health checks for SQL Server docker service Remove `!` from password so `SA_PASSWORD` doesn't need extra quotes. * Fix install_mssql.sh commands * Use `isolation_level` argument also for `mssql` * Restore `engine.dispose()` * Add `coverage.xml` to `.gitignore` * change tests to use non-deprecated class Change EncryptedType tests to use StringEncryptedType * use scalar subquery for sqlalchemy 1.4 * use create_mock_engine for SA 1.4 * isort linting fixes Co-authored-by: Matthias Bernt <m.bernt@ufz.de> Co-authored-by: Srinivasan R <srinivasanr@gmail.com> Co-authored-by: Luca Venturini <luca.venturini@earlham.ac.uk> Co-authored-by: Braden Mars <bradenmars@bradenmars.me> Co-authored-by: Konsta Vesterinen <konsta.vesterinen@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
database_exists()on PostgreSQL. Fix 0.36.8 breaks database_exists() call #462 , supersedes fix database existence check #463create_databasesupportpymssql#486 , supersedes Fix psycopg2 issues with database creation #494