Synapse does not take advantage of Pydantic 2.0 #15858

c7hm4r · 2023-07-01T15:09:27Z

Description

After upgrading the PIP packages, Synapse failed to launch.

Steps to reproduce

install matrix-synapse using the "virtualenv" approach with the most recent dependency versions
start Synapse

Homeserver

another homeserver

Synapse Version

1.87.0rc1

Installation Method

pip (from PyPI)

Database

PostgreSQL

Workers

Single process

Platform

Raspberry Pi OS (Debian)

Configuration

No response

Relevant log output

/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/_internal/_config.py:257: UserWarning: Valid config keys have changed in V2:                                                            
* 'allow_mutation' has been removed                                                
  warnings.warn(message, UserWarning)                                              
Traceback (most recent call last):                                                 
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main             
    return _run_code(code, main_globals, None,                                     
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code                        
    exec(code, run_globals)                                                        
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/homeserver.py", line 37, in <module>                                                                                         
    from synapse.app import _base                                                  
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/_base.py", line 68, in <module>                                                                                              
    from synapse.handlers.auth import load_legacy_password_auth_providers          
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/auth.py", line 56, in <module>                                                                                          
    from synapse.api.ratelimiting import Ratelimiter                              
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/api/ratelimiting.py", line 21, in <module>                                                                                       
    from synapse.storage.databases.main import DataStore                           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/__init__.py", line 34, in <module>                                                                                       
    from synapse.storage.databases import Databases                                
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/__init__.py", line 20, in <module>                                                                             
    from synapse.storage.databases.main.events import PersistEventsStore           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/__init__.py", line 27, in <module>                                                                        
    from synapse.storage.databases.main.stats import UserSortOrder                 
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/stats.py", line 42, in <module>                                                                           
    from synapse.storage.databases.main.events_worker import InvalidEventError     
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_worker.py", line 62, in <module>
    from synapse.replication.tcp.streams import BackfillStream, UnPartialStatedEventStream
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/__init__.py", line 27, in <module>
    from synapse.replication.tcp.streams._base import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/_base.py", line 32, in <module>
    from synapse.replication.http.streams import ReplicationGetStreamUpdates
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/__init__.py", line 18, in <module>
    from synapse.replication.http import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/account_data.py", line 21, in <module>
    from synapse.replication.http._base import ReplicationEndpoint
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/_base.py", line 31, in <module>
    from synapse.http.servlet import parse_json_object_from_request
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/http/servlet.py", line 32, in <module>
    from pydantic import BaseModel, MissingError, PydanticValueError, ValidationError
ImportError: cannot import name 'MissingError' from 'pydantic' (/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/__init__.py)
systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

Anything else that would be useful to know?

pydantic 2.0 was released yesterday

The text was updated successfully, but these errors were encountered:

c7hm4r · 2023-07-01T15:10:14Z

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

DMRobertson · 2023-07-02T09:49:46Z

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

Suggest we specify this in pyproject.toml, or change

synapse/pyproject.toml

Lines 209 to 210 in 07fd6d8

    
           # This is the most recent version of Pydantic with available on common distros. 
        
           pydantic = ">=1.7.4"

to ^=1.7.4.

Not sure why I didn't do this when starting to use Pydantic. I think because generally speaking it makes packagers' lives easier if we don't impose upper bounds?

PaarthShah · 2023-07-02T20:37:08Z

Just encountered this, and in other projects as well on Friday morning. I agree with pinning to <2.0.0 as a workaround; hoping to make a PR for this shortly.

PaarthShah · 2023-07-02T20:54:36Z

Just created: #15862

Pillow 10 removes some deprecated constatns that are used in synapse, the issue is already fixed in upstream, see [1,2]. Moreover, synapse is incompatible with pydantic 2.0 [3]. [1] matrix-org/synapse#15873 [2] matrix-org/synapse#15876 [3] matrix-org/synapse#15858 Closes: https://bugs.gentoo.org/909644 Signed-off-by: Petr Vaněk <arkamar@atlas.cz>

H-Shay · 2023-07-05T21:13:36Z

Closing this as it looks like it is resolved by #15862

Pillow 10 removes some deprecated constatns that are used in synapse, the issue is already fixed in upstream, see [1,2]. Moreover, synapse is incompatible with pydantic 2.0 [3]. [1] matrix-org/synapse#15873 [2] matrix-org/synapse#15876 [3] matrix-org/synapse#15858 Closes: https://bugs.gentoo.org/909644 Signed-off-by: Petr Vaněk <arkamar@atlas.cz> Signed-off-by: Sam James <sam@gentoo.org>

dvzrv · 2023-07-23T11:33:59Z

Hi! We're currently blocked by matrix-synapse in our pydantic 2 rebuild for Arch Linux. Please reopen this ticket so that discussion around how/when/if to switch to pydantic2 can happen.

The change in #15862 is merely a workaround and not the solution to this ticket (which would be "be compatible with pydantic >= 2").

Thanks! :)

Adapt codebase to use pydantic >= 2 models and functionalities. Remove unneeded checks from `scripts-dev/check_pydantic_models.py`, since pydantic can now be used in a strict mode which will prevent the type coercion: https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode Closes matrix-org#15858

Adapt codebase to use pydantic >= 2 models and functionalities. Remove unneeded checks from `scripts-dev/check_pydantic_models.py`, since pydantic can now be used in a strict mode which will prevent the type coercion: https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode Closes matrix-org#15858 Signed-off-by: David Runge <dave@sleepmap.de>

clokep · 2023-07-24T11:49:14Z

Re-opening per the above -- pinning to <= 2.0 isn't supporting Pydantic 2.0.

Adapt codebase to use pydantic >= 2 models and functionalities. Remove unneeded checks from `scripts-dev/check_pydantic_models.py`, since pydantic can now be used in a strict mode which will prevent the type coercion: https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode Closes matrix-org#15858 Signed-off-by: David Runge <dave@sleepmap.de>

reivilibre · 2023-08-07T15:18:42Z

Pydantic V2 includes V1 built-in to help migration.

We could use the following all over the place as a transitional measure.

try:
    from pydantic import v1 as pydantic
except ImportError:
    import pydantic

Given that V1 and V2 use the same Python package name (⇒ you can't have them both installed at the same time), we sadly can expect some trouble with distro packages that would make switching to using V2 proper difficult for some time.

dvzrv · 2023-08-13T10:20:44Z

We could use the following all over the place as a transitional measure.

I'm not sure if that really works as intended. There are some implied breakages AFAIK irt the validators.

I can earliest look at my open PR again after 2023-08-21. If someone manages to fix this earlier and create a new release, that would be much appreciated, as synapse is currently the only package blocking us from moving to pydantic v2 on Arch Linux: https://archlinux.org/todo/python-pydantic-2/

@PaarthShah

Please note that this will be the last release of Synapse that is compatible with Python 3.7 and earlier. This is due to Python 3.7 now having reached End of Life; see our [deprecation policy](https://matrix-org.github.io/synapse/v1.87/deprecation_policy.html) for more details. - Pin `pydantic` to `^1.7.4` to avoid backwards-incompatible API changes from the 2.0.0 release. Resolves matrix-org#15858. Contributed by @PaarthShah. ([\matrix-org#15862](matrix-org#15862)) - Split out 2022 changes from the changelog so the rendered version in GitHub doesn't timeout as much. ([\matrix-org#15846](matrix-org#15846)) - Improve `/messages` response time by avoiding backfill when we already have messages to return. ([\matrix-org#15737](matrix-org#15737)) - Add spam checker module API for logins. ([\matrix-org#15838](matrix-org#15838)) - Fix a long-standing bug where media files were served in an unsafe manner. Contributed by @joshqou. ([\matrix-org#15680](matrix-org#15680)) - Avoid invalidating a cache that was just prefilled. ([\matrix-org#15758](matrix-org#15758)) - Fix requesting multiple keys at once over federation, related to [MSC3983](matrix-org/matrix-spec-proposals#3983). ([\matrix-org#15770](matrix-org#15770)) - Fix joining rooms through aliases where the alias server isn't a real homeserver. Contributed by @tulir @ Beeper. ([\matrix-org#15776](matrix-org#15776)) - Fix a bug in push rules handling leading to an invalid (per spec) `is_user_mention` rule sent to clients. Also fix wrong rule names for `is_user_mention` and `is_room_mention`. ([\matrix-org#15781](matrix-org#15781)) - Fix a bug introduced in 1.57.0 where the wrong table would be locked on updating database rows when using SQLite as the database backend. ([\matrix-org#15788](matrix-org#15788)) - Fix Sytest environmental variable evaluation in CI. ([\matrix-org#15804](matrix-org#15804)) - Fix forgotten rooms missing from initial sync after rejoining them. Contributed by Nico from Famedly. ([\matrix-org#15815](matrix-org#15815)) - Fix sqlite `user_filters` upgrade introduced in v1.86.0. ([\matrix-org#15817](matrix-org#15817)) - Document `looping_call()` functionality that will wait for the given function to finish before scheduling another. ([\matrix-org#15772](matrix-org#15772)) - Fix a typo in the [Admin API](https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/index.html). ([\matrix-org#15805](matrix-org#15805)) - Fix typo in MSC number in faster remote room join architecture doc. ([\matrix-org#15812](matrix-org#15812)) - Remove experimental [MSC2716](matrix-org/matrix-spec-proposals#2716) implementation to incrementally import history into existing rooms. ([\matrix-org#15748](matrix-org#15748)) - Replace `EventContext` fields `prev_group` and `delta_ids` with field `state_group_deltas`. ([\matrix-org#15233](matrix-org#15233)) - Regularly try to send transactions to other servers after they failed instead of waiting for a new event to be available before trying. ([\matrix-org#15743](matrix-org#15743)) - Fix requesting multiple keys at once over federation, related to [MSC3983](matrix-org/matrix-spec-proposals#3983). ([\matrix-org#15755](matrix-org#15755)) - Allow for the configuration of max request retries and min/max retry delays in the matrix federation client. ([\matrix-org#15783](matrix-org#15783)) - Switch from `matrix://` to `matrix-federation://` scheme for internal Synapse routing of outbound federation traffic. ([\matrix-org#15806](matrix-org#15806)) - Fix harmless exceptions being printed when running the port DB script. ([\matrix-org#15814](matrix-org#15814)) * Bump attrs from 22.2.0 to 23.1.0. ([\matrix-org#15801](matrix-org#15801)) * Bump cryptography from 40.0.2 to 41.0.1. ([\matrix-org#15800](matrix-org#15800)) * Bump ijson from 3.2.0.post0 to 3.2.1. ([\matrix-org#15802](matrix-org#15802)) * Bump phonenumbers from 8.13.13 to 8.13.14. ([\matrix-org#15798](matrix-org#15798)) * Bump ruff from 0.0.265 to 0.0.272. ([\matrix-org#15799](matrix-org#15799)) * Bump ruff from 0.0.272 to 0.0.275. ([\matrix-org#15833](matrix-org#15833)) * Bump serde_json from 1.0.96 to 1.0.97. ([\matrix-org#15797](matrix-org#15797)) * Bump serde_json from 1.0.97 to 1.0.99. ([\matrix-org#15832](matrix-org#15832)) * Bump towncrier from 22.12.0 to 23.6.0. ([\matrix-org#15831](matrix-org#15831)) * Bump types-opentracing from 2.4.10.4 to 2.4.10.5. ([\matrix-org#15830](matrix-org#15830)) * Bump types-setuptools from 67.8.0.0 to 68.0.0.0. ([\matrix-org#15835](matrix-org#15835))

Adapt codebase to use pydantic >= 2 models and functionalities. Remove unneeded checks from `scripts-dev/check_pydantic_models.py`, since pydantic can now be used in a strict mode which will prevent the type coercion: https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode Closes matrix-org#15858 Signed-off-by: David Runge <dave@sleepmap.de>

gotmax23 · 2023-09-11T22:39:00Z

We could use the following all over the place as a transitional measure.

I'm not sure if that really works as intended. There are some implied breakages AFAIK irt the validators.

The pydantic.v1 module is simply a copy of the old pydantic v1 code, so there should not be breakages there.

gotmax23 · 2023-09-17T04:39:11Z

I opened #16332 to add support for pydantic v2 while maintaining support for v1.

clokep · 2023-10-02T14:28:42Z

I opened #16332 to add support for pydantic v2 while maintaining support for v1.

I re-titled this since Synapse is now compatible, but not taking advantage of pydantic v2.

DMRobertson added X-Release-Blocker Must be resolved before making a release X-Regression Something broke which worked on a previous release labels Jul 2, 2023

michaelkaye mentioned this issue Jul 3, 2023

ImportError when using this action michaelkaye/setup-matrix-synapse#94

Closed

poljar mentioned this issue Jul 3, 2023

test: Change the installer for setup-matrix-synapse matrix-org/matrix-rust-sdk#2198

Merged

H-Shay closed this as completed Jul 5, 2023

dvzrv linked a pull request Jul 23, 2023 that will close this issue

Update pydantic to >= 2 #15979

Open

4 tasks

clokep reopened this Jul 24, 2023

MatMaul removed X-Release-Blocker Must be resolved before making a release X-Regression Something broke which worked on a previous release labels Jul 25, 2023

clokep added S-Minor Blocks non-critical functionality, workarounds exist. T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. O-Occasional Affects or can be seen by some users regularly or most users rarely labels Jul 25, 2023

DMRobertson mentioned this issue Sep 4, 2023

Bump pydantic from 1.10.11 to 2.3.0 #16234

Closed

erikjohnston mentioned this issue Sep 11, 2023

Bump pydantic from 1.10.11 to 2.2.1 #16294

Closed

polyzen mentioned this issue Sep 17, 2023

from pydantic import v1 as pydantic #16331

Closed

4 tasks

gotmax23 mentioned this issue Sep 17, 2023

Add support for pydantic v2 via pydantic.v1 compat module #16332

Merged

4 tasks

clokep changed the title ~~Synapse is incompatible with Pydantic 2.0~~ Synapse does not take advantage of Pydantic 2.0 Oct 2, 2023

matrixbot mentioned this issue Dec 22, 2023

Synapse does not take advantage of Pydantic 2.0 element-hq/synapse#15858

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Synapse does not take advantage of Pydantic 2.0 #15858

Synapse does not take advantage of Pydantic 2.0 #15858

c7hm4r commented Jul 1, 2023

c7hm4r commented Jul 1, 2023

DMRobertson commented Jul 2, 2023

PaarthShah commented Jul 2, 2023 •

edited

Loading

PaarthShah commented Jul 2, 2023 •

edited

Loading

H-Shay commented Jul 5, 2023

dvzrv commented Jul 23, 2023

clokep commented Jul 24, 2023

reivilibre commented Aug 7, 2023

dvzrv commented Aug 13, 2023

gotmax23 commented Sep 11, 2023

gotmax23 commented Sep 17, 2023

clokep commented Oct 2, 2023

Synapse does not take advantage of Pydantic 2.0 #15858

Synapse does not take advantage of Pydantic 2.0 #15858

Comments

c7hm4r commented Jul 1, 2023

Description

Steps to reproduce

Homeserver

Synapse Version

Installation Method

Database

Workers

Platform

Configuration

Relevant log output

Anything else that would be useful to know?

c7hm4r commented Jul 1, 2023

DMRobertson commented Jul 2, 2023

PaarthShah commented Jul 2, 2023 • edited Loading

PaarthShah commented Jul 2, 2023 • edited Loading

H-Shay commented Jul 5, 2023

dvzrv commented Jul 23, 2023

clokep commented Jul 24, 2023

reivilibre commented Aug 7, 2023

dvzrv commented Aug 13, 2023

gotmax23 commented Sep 11, 2023

gotmax23 commented Sep 17, 2023

clokep commented Oct 2, 2023

PaarthShah commented Jul 2, 2023 •

edited

Loading

PaarthShah commented Jul 2, 2023 •

edited

Loading