Some rooms do not get sent down via sliding sync; rooms impacted differ per account

[composite9239]

Description

Leaving and rejoining a room causes a new login on Element X to not receive the rooms via sliding sync.

Steps to reproduce

• Log into both Element Desktop and Element X on @joe:example.com. @joe:example.com is joined to room 1, room 2, room 3, room 4, and room 5. All rooms are visible on Element Desktop and Element X.
• Joe logs out from his account on Element X.
• Joe is kicked from room 2 and leaves room 3 of his own volition. (It makes no difference)
• Joe rejoins room 2 and room 3.
• Joe logs in on Element X and room 2 and room 3 are invisible because they are not sent down via sliding sync, so he can only see rooms 1, 4, and 5.
• As a result, Joe can be mentioned in room 2 and room 3, which will display on desktop but Element X will act as though he has no unreads because it doesn't receive the rooms.

It was confirmed via Element X logs that the rooms don't get sent down via sliding sync.

Homeserver

another homeserver

Synapse Version

1.115.0

Installation Method

Other (please mention below)

Database

PostgreSQL 17.0

Workers

Multiple workers

Platform

Ubuntu 22.04.5 LTS

Configuration

No response

Relevant log output

Unsure of relevant portion, but this should be reproducible

Anything else that would be useful to know?

using matrix-docker-ansible-deploy playbook with specialized-workers preset

native sliding sync (no proxy)

[erikjohnston]

Hi, that's deeply concerning.

I'd be interested to know the outcomes of the following queries, with the room ID and user ID of the room/user in question:

SELECT * FROM background_updates;

SELECT sender != user_id, membership, room_type FROM sliding_sync_membership_snapshots WHERE room_id = ? AND user_id = ?

SELECT room_id, room_type FROM sliding_sync_joined_rooms WHERE room_id = ?

SELECT membership FROM local_current_membership WHERE room_id = ? AND user_id = ?

[composite9239]

Hi, that's deeply concerning.

I'd be interested to know the outcomes of the following queries, with the room ID and user ID of the room/user in question:

SELECT * FROM background_updates;

SELECT sender != user_id, membership, room_type FROM sliding_sync_membership_snapshots WHERE room_id = ? AND user_id = ?

SELECT room_id, room_type FROM sliding_sync_joined_rooms WHERE room_id = ?

SELECT membership FROM local_current_membership WHERE room_id = ? AND user_id = ?

Hello, here are the responses to those queries.

1. (0 rows)
2. ?column? says f, membership says join, room_type is blank (1 row)
3. reprints the room ID under room_id, room_type is blank (1 row)
4. under "membership" says "join" (1 row)

[erikjohnston]

Huh, so looks like the DB is in a sane state (I was worried that there might have been a problem with the background update).

I'd be interested in knowing if the room appeared in the responses to the following requests (you'll need to get an access token from the access_tokens table), replacing the appropriate values:

curl 'https://<HOMESERVER>/_matrix/client/unstable/org.matrix.simplified_msc3575/sync' -H "Authorization: Bearer <ACCESS_TOKEN>" -X POST -d '{"conn_id": "test", "room_subscriptions": {"<ROOM_ID>": {"timeline_limit": 0, "required_state": []}}}'

[composite9239]

It gives me two different kinds of responses depending on the room I do it for.

When I do it for a room that does not show up on Element X for that account, it gives the following:

"pos": "0/[long string of numbers/underscores]"
"rooms": {}
"extensions": {}

When I do it for a room that does show up on Element X for that account, it gives me what I'm guessing is expected output, including pos ([a nonzero number]/[long string of numbers/underscores]), rooms {bump_stamp, notification_count, highlight_count, joined_count, invited_count, name, avatar, initial}, extensions: {}

[erikjohnston]

Interesting, interesting. So the room isn't being incorrectly filtered out by the room lists.

Can you run (to see if it returns anything):

SELECT m.room_id, m.sender, m.membership, m.membership_event_id,
                    r.room_version,
                    m.event_instance_name, m.event_stream_ordering,
                    m.has_known_state,
                    COALESCE(j.room_type, m.room_type),
                    COALESCE(j.is_encrypted, m.is_encrypted)
                FROM sliding_sync_membership_snapshots AS m
                INNER JOIN rooms AS r USING (room_id)
                LEFT JOIN sliding_sync_joined_rooms AS j ON (j.room_id = m.room_id AND m.membership = 'join')
                WHERE user_id = ?
                    AND m.forgotten = 0
                    AND m.room_id = ?

And:

SELECT room_version FROM rooms WHERE room_id = ?

SELECT membership, forgotten FROM room_memberships WHERE room_id = ? AND user_id = ?

These are the queries that pull the room info from the DB, so should show us if its not returning the room for some reason and if so why.

[composite9239]

1. 0 rows (shows room_id | sender | membership | etc columns)
2. room_version -> 10
3. room_memberships table:

 membership | forgotten 
------------+-----------
 invite     |         1
 join       |         0
 leave      |         1
 invite     |         0
 join       |         1
 leave      |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
(15 rows)

Here's step 3 again for another room that doesn't show up on Element X (this one is room version 9, not 10):

 membership | forgotten 
------------+-----------
 join       |         1
 join       |         1
 leave      |         1
 join       |         0
(4 rows)

And here's step 3 again for a room that does show up on Element X (room version 10):

 membership | forgotten 
------------+-----------
 join       |         0
 join       |         0
 join       |         0
 join       |         0
 join       |         0
 join       |         0
 join       |         0
 join       |         0
 join       |         0
(9 rows)

Here are some possibly relevant configuration options we have set:

auto_accept_invites_enabled: true

auto_accept_invites_only_from_local_users: true

forget_rooms_on_leave: true

block_non_admin_invites: true

alias_creation_rules:
  - user_id: "*"
    alias: "*"
    room_id: "*"
    action: deny

room_list_publication_rules:
  - user_id: "*"
    alias: "*"
    room_id: "*"
    action: deny

limit_remote_rooms:
  enabled: true
  complexity: 0.0
  admins_can_join: true

With regard to the above, it should be noted that the issue happens for both Synapse admin and non-Synapse-admin accounts.

[erikjohnston]

Aha, looks like sliding_sync_membership_snapshots thinks the room is forgotten, would be my guess. Final SQL queries hopefully:

SELECT membership_event_id, membership, forgotten
FROM sliding_sync_membership_snapshots
WHERE room_id = ? AND user_id = ?

SELECT event_id, membership, forgotten
FROM room_memberships AS m
INNER JOIN events USING (event_id)
WHERE m.room_id = ? AND m.user_id = ?
ORDER BY events.depth ASC

SELECT event_id, membership
FROM local_current_membership
WHERE room_id = ? AND user_id = ?

The idea being that we want to see if a) all the event IDs match with the same membership, and b) if sliding_sync_membership_snapshots does indeed incorrectly have forgotten set

[composite9239]

1. Gives a membership event ID, says "join" under "membership," and "1" under "forgotten".
2. Omitting the event IDs, it returns:

 membership | forgotten 
------------+-----------
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 join       |         1
 leave      |         1
 invite     |         1
 join       |         1
 leave      |         1
 invite     |         0
 join       |         0 (for this one the event ID is the same one as earlier)

3. Gives the same event ID as earlier and under "membership" says "join".

We have forget_rooms_on_leave set to true, if that makes a difference.

[erikjohnston]

OK great. That's the problem then, somehow the forgotten flag is set in sliding_sync_membership_snapshots when it shouldn't be (as that membership event ID wasn't forgotten).

[MadLittleMods]

Thanks for the report @composite9239! The fix will be included in the next upcoming release and existing rooms affected by the bug will automatically be fixed by a background update that will run after you upgrade.

[composite9239]

Excellent, thank you very much!