ERROR - POST-758856 - Failed handle request via 'RoomMembershipRestServlet' – when trying to leave room #13524

schildbach · 2022-08-15T09:21:52Z

Description

I'm trying to leave the "Matrix Community→Servers/Hosting→Synapse Admins" room (!ehXvUhWNASUkSLvAGP:matrix.org) via one of my clients (doesn't matter). All I get is Internal server error on the client, and

synapse_1     | 2022-08-15 11:50:10,254 - synapse.http.server - 183 - ERROR - POST-776070 - Failed handle request via 'RoomMembershipRestServlet': <XForwardedForRequest at 0x7fb66c5eedf0 method='POST' uri='/_matrix/client/r0/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/leave' clientproto='HTTP/1.1' site='8008'>
synapse_1     | Traceback (most recent call last):
synapse_1     |   File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
synapse_1     |     result = current_context.run(gen.send, result)
synapse_1     | StopIteration: []
synapse_1     | 
synapse_1     | During handling of the above exception, another exception occurred:
synapse_1     | 
synapse_1     | Traceback (most recent call last):
synapse_1     |   File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
synapse_1     |     result = current_context.run(gen.send, result)
synapse_1     |   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/state.py", line 371, in _get_state_group_for_events
synapse_1     |     raise RuntimeError("No state group for unknown or outlier event %s" % e)
synapse_1     | RuntimeError: No state group for unknown or outlier event $1jY0NigFw_kN3uj5T5KbpXHJCXaB_5ds1bIvLJHW9Hc

on the server (Synapse 1.64.0).

I tried clearing the client caches (didn't help).

I'd expect to be able to leave the room under any circumstances.

Steps to reproduce

Just try to leave the room.

Homeserver

matrix.schildbach.de

Synapse Version

1.64.0

Installation Method

Docker image

Platform

Docker host is Ubuntu 22.04.1 all pretty much stock.

Relevant log output

synapse_1     | 2022-08-15 11:50:10,254 - synapse.http.server - 183 - ERROR - POST-776070 - Failed handle request via 'RoomMembershipRestServlet': <XForwardedForRequest at 0x7fb66c5eedf0 method='POST' uri='/_matrix/client/r0/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/leave' clientproto='HTTP/1.1' site='8008'>
synapse_1     | Traceback (most recent call last):
synapse_1     |   File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
synapse_1     |     result = current_context.run(gen.send, result)
synapse_1     | StopIteration: []
synapse_1     | 
synapse_1     | During handling of the above exception, another exception occurred:
synapse_1     | 
synapse_1     | Traceback (most recent call last):
synapse_1     |   File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
synapse_1     |     result = current_context.run(gen.send, result)
synapse_1     |   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/state.py", line 371, in _get_state_group_for_events
synapse_1     |     raise RuntimeError("No state group for unknown or outlier event %s" % e)
synapse_1     | RuntimeError: No state group for unknown or outlier event $1jY0NigFw_kN3uj5T5KbpXHJCXaB_5ds1bIvLJHW9Hc

Anything else that would be useful to know?

More log output, grepped by the room ID, attached as file.

log.txt

The text was updated successfully, but these errors were encountered:

DMRobertson · 2022-08-15T10:37:12Z

More log output, grepped by the room ID, attached as file.

We're going to need more logs to diagnose this. Can you grep for POST-758856 and provide those logs?

schildbach · 2022-08-15T11:21:39Z

We're going to need more logs to diagnose this. Can you grep for POST-758856 and provide those logs?

It only yields the line I already posted:

# docker-compose logs | grep "POST-758856"
synapse_1     | 2022-08-15 09:10:46,431 - synapse.http.server - 183 - ERROR - POST-758856 - Failed handle request via 'RoomMembershipRestServlet': <XForwardedForRequest at 0x7fb5d963e460 method='POST' uri='/_matrix/client/r0/rooms/!ehXvUhWNASUkSLvAGP:matrix.org/leave' clientproto='HTTP/1.1' site='8008'>

DMRobertson · 2022-08-15T11:28:24Z

Are there any lines containing the word Traceback in the logs? Perhaps you could attach the last minutes' worth of logs prior to the error you mention.

schildbach · 2022-08-15T11:30:54Z

I see for example this traceback, but it's for a different room it seems:

ESC[33msynapse_1     |ESC[0m 2022-08-15 11:20:06,316 - synapse.handlers.federation_event - 1421 - INFO - _process_incoming_pdus_in_room_inner-5027-$BzPBUE0Lvw60fy8IPD5f9uX4I
b4A_n4kYjgqxYj1auU-$oCN4fyrk3iGEl1beYLt03h2FfFASSzgA3y8nvwmOKxQ-$vzKvob-AWdICbGLNEmKpgZ8Ci1G4LddZurGz3kQ5-uM - Fetched 17 events of 17 requested
ESC[33msynapse_1     |ESC[0m 2022-08-15 11:20:06,319 - synapse.handlers.federation_event - 1474 - INFO - _process_incoming_pdus_in_room_inner-5027-$BzPBUE0Lvw60fy8IPD5f9uX4I
b4A_n4kYjgqxYj1auU-$oCN4fyrk3iGEl1beYLt03h2FfFASSzgA3y8nvwmOKxQ-$vzKvob-AWdICbGLNEmKpgZ8Ci1G4LddZurGz3kQ5-uM - Persisting 15 of 17 remaining outliers: ['$RSMVfJFHrfO_4XItQox
ZA3xCuakVg2QA6kALZyfpG3U', '$4Re_XqEigZ6Ameuc778RImAkMjmkrnYR6tM9kjQ-x0w', '$F-dcd_eX0eoFxbKebuqCubY7gkyML7ZtgIvDA-LjCLw', '$DDPpt3Mrjk37tNlrKBJz3Y6kcXw-5H67jRXz6noUklw', '$
--tyOF84611zAXloqauxgHFHbG7lGQd-BLf2p9nGVB0', ...]
ESC[33msynapse_1     |ESC[0m 2022-08-15 11:20:06,347 - synapse.handlers.federation_event - 1474 - INFO - _process_incoming_pdus_in_room_inner-5027-$BzPBUE0Lvw60fy8IPD5f9uX4I
b4A_n4kYjgqxYj1auU-$oCN4fyrk3iGEl1beYLt03h2FfFASSzgA3y8nvwmOKxQ-$vzKvob-AWdICbGLNEmKpgZ8Ci1G4LddZurGz3kQ5-uM - Persisting 2 of 2 remaining outliers: ['$q9_7Br3lp1BHDtbdFlGQJ
MqDtpz_O4Oz-J2jElnbe1M', '$00LZP1sLA1AfeO-wHxvyKo0Jhd8CA7QgCnVmrD_bwwk']
ESC[33msynapse_1     |ESC[0m 2022-08-15 11:20:07,855 - synapse.federation.federation_server - 1184 - ERROR - _process_incoming_pdus_in_room_inner-5027 - Failed to handle PDU
 $BzPBUE0Lvw60fy8IPD5f9uX4Ib4A_n4kYjgqxYj1auU
ESC[33msynapse_1     |ESC[0m Traceback (most recent call last):
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
ESC[33msynapse_1     |ESC[0m     result = current_context.run(gen.send, result)
ESC[33msynapse_1     |ESC[0m StopIteration: []
ESC[33msynapse_1     |ESC[0m 
ESC[33msynapse_1     |ESC[0m During handling of the above exception, another exception occurred:
ESC[33msynapse_1     |ESC[0m 
ESC[33msynapse_1     |ESC[0m Traceback (most recent call last):
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 239, in on_receive_pdu
ESC[33msynapse_1     |ESC[0m     await self._get_missing_events_for_pdu(
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 705, in _get_missing_events_for_pdu
ESC[33msynapse_1     |ESC[0m     await self._process_pulled_events(origin, missing_events, backfilled=False)
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 743, in _process_pulled_events
ESC[33msynapse_1     |ESC[0m     await self._process_pulled_event(origin, ev, backfilled=backfilled)
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 814, in _process_pulled_event
ESC[33msynapse_1     |ESC[0m     await self._process_received_pdu(
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 1147, in _process_received_pdu
ESC[33msynapse_1     |ESC[0m     await self._check_for_soft_fail(event, state_ids, origin=origin)
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 1758, in _check_for_soft_fail
ESC[33msynapse_1     |ESC[0m     state_sets_d = await self._state_storage_controller.get_state_groups_ids(
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/storage/controllers/state.py", line 103, in get_state_groups_ids
ESC[33msynapse_1     |ESC[0m     event_to_groups = await self.get_state_group_for_events(event_ids)
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/storage/controllers/state.py", line 341, in get_state_group_for_events
ESC[33msynapse_1     |ESC[0m     return await self.stores.main._get_state_group_for_events(event_ids)
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
ESC[33msynapse_1     |ESC[0m     result = current_context.run(gen.send, result)
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/state.py", line 371, in _get_state_group_for_events
ESC[33msynapse_1     |ESC[0m     raise RuntimeError("No state group for unknown or outlier event %s" % e)
ESC[33msynapse_1     |ESC[0m RuntimeError: No state group for unknown or outlier event $1jY0NigFw_kN3uj5T5KbpXHJCXaB_5ds1bIvLJHW9Hc
ESC[33msynapse_1     |ESC[0m 
ESC[33msynapse_1     |ESC[0m The above exception was the direct cause of the following exception:
ESC[33msynapse_1     |ESC[0m 
ESC[33msynapse_1     |ESC[0m Traceback (most recent call last):
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/federation/federation_server.py", line 1174, in _process_incoming_pdus_in_room_inner
ESC[33msynapse_1     |ESC[0m     await self._federation_event_handler.on_receive_pdu(
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 243, in on_receive_pdu
ESC[33msynapse_1     |ESC[0m     raise Exception(
ESC[33msynapse_1     |ESC[0m Exception: Error fetching missing prev_events for $BzPBUE0Lvw60fy8IPD5f9uX4Ib4A_n4kYjgqxYj1auU: No state group for unknown or outlier event $1jY
0NigFw_kN3uj5T5KbpXHJCXaB_5ds1bIvLJHW9Hc
ESC[33msynapse_1     |ESC[0m 2022-08-15 11:20:08,381 - synapse.metrics._gc - 120 - INFO - sentinel - Collecting gc 1

schildbach · 2022-08-15T11:37:40Z

This one seems related:

ESC[33msynapse_1     |ESC[0m 2022-08-15 11:34:48,587 - synapse.http.server - 183 - ERROR - POST-774533 - Failed handle request via 'RoomMembershipRestServlet': <XForwardedForRequest at 0x7fb6734929a0 method='POST' uri='/_matrix/client/r0/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/leave' clientproto='HTTP/1.1' site='8008'>
ESC[33msynapse_1     |ESC[0m Traceback (most recent call last):
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
ESC[33msynapse_1     |ESC[0m     result = current_context.run(gen.send, result)
ESC[33msynapse_1     |ESC[0m StopIteration: []
ESC[33msynapse_1     |ESC[0m 
ESC[33msynapse_1     |ESC[0m During handling of the above exception, another exception occurred:
ESC[33msynapse_1     |ESC[0m 
ESC[33msynapse_1     |ESC[0m Traceback (most recent call last):
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
ESC[33msynapse_1     |ESC[0m     result = current_context.run(gen.send, result)
ESC[33msynapse_1     |ESC[0m   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/state.py", line 371, in _get_state_group_for_events
ESC[33msynapse_1     |ESC[0m     raise RuntimeError("No state group for unknown or outlier event %s" % e)
ESC[33msynapse_1     |ESC[0m RuntimeError: No state group for unknown or outlier event $1jY0NigFw_kN3uj5T5KbpXHJCXaB_5ds1bIvLJHW9Hc
(end of log at this time)

schildbach · 2022-08-15T11:43:05Z

Maybe I should mention that I am (or was, not sure) subscribed to !ehXvUhWNASUkSLvAGP:matrix.org via the "Servers/Hosting" space, via the "Matrix Community" space. I don't know how to find out the "internal room id" of spaces in the element client. Spaces are missing the "Settings → Advanced" dialog.

DMRobertson · 2022-08-15T12:11:59Z

What kind of database are you running Synapse against: sqlite or postgres? If postgres, have you recently migrated from sqlite? Also, do you have any experimental options turned on in config?

schildbach · 2022-08-15T13:31:30Z

What kind of database are you running Synapse against: sqlite or postgres? If postgres, have you recently migrated from sqlite? Also, do you have any experimental options turned on in config?

I'm using postgres 14.2, have migrated from sqlite 3-4 years ago, I think well before I joined the group. No experimental at all in homeserver.yaml or postgresql.conf, I always try to stay with defaults as much as possible.

richvdh · 2022-08-15T17:47:14Z

do you have message retention enabled? Or have you used one of the admin APIs to delete messages from this room?

schildbach · 2022-08-15T18:26:07Z

I have these retention policies:

retention:
  enabled: true
  default_policy:
    min_lifetime: 1d
    max_lifetime: 26w

media_retention:
    local_media_lifetime: 180d
    remote_media_lifetime: 30d

schildbach · 2022-08-15T18:27:07Z

I'm not aware of having deleted message via the admin API. I do purge rooms without local subscribers though, supposedly including their messages.

richvdh · 2022-08-15T18:27:43Z

I have these retention policies:

That'll do it. Duplicate #13476.

schildbach · 2022-08-15T18:29:48Z

I assume I should asap disable message retention then, until the issue is fixed? How can I recover from the corrupted database?

richvdh · 2022-08-15T18:32:35Z

I assume I should asap disable message retention then, until the issue is fixed?

Best to.

How can I recover from the corrupted database?

Most reliable mechanism is to completely purge the room (with "force_purge": true if necessary). Then if you want to rejoin the room, restart synapse before doing so.

Sorry about the mess.

schildbach · 2022-08-15T18:59:07Z

I'm not sure if this worked:

$ curl --header "Authorization: Bearer xxx" -X DELETE -H "Content-Type: application/json" -d "{ \"purge\": true, \"force_purge\": true }" 'https://matrix.schildbach.de/_synapse/admin/v1/rooms/!ehXvUhWNASUkSLvAGP:matrix.org'
{"kicked_users":[],"failed_to_kick_users":["@andreas:schildbach.de"],"local_aliases":[],"new_room_id":null}

schildbach · 2022-08-15T19:04:05Z

…but indeed despite me being listed under "failed_to_kick_users" it kicked me from the room. Thanks for the help!

rettichschnidi · 2022-08-18T13:44:01Z

Most reliable mechanism is to completely purge the room (with "force_purge": true if necessary). Then if you want to rejoin the room, restart synapse before doing so.

Tried this, but it does not seem to actually delete most of the entries in table state_groups_state:

Number of entries before deletion:

synapse=# SELECT count(s.room_id) FROM state_groups_state s WHERE room_id='!mjbDjyNsRXndKLkHIe:matrix.org';
 count
--------
 803976
(1 row)

Delete room (old version of #synapse:matrix.org in this case):

curl 'http://localhost:8008/_synapse/admin/v1/rooms/!mjbDjyNsRXndKLkHIe:matrix.org' -X DELETE -H 'authorization: Bearer XXX' -H 'content-type: application/json' --data '{ "force_purge": true }'

Resulting logs:

root@matrix-1:~# journalctl --since "15 min ago" | grep DELETE-2262217
Aug 18 13:25:40 matrix-1 synapse[2542335]: synapse.handlers.room: [DELETE-2262217] Shutting down room '!mjbDjyNsRXndKLkHIe:matrix.org'
Aug 18 13:25:40 matrix-1 synapse[2542335]: synapse.handlers.room: [DELETE-2262217] Kicking '@rettichschnidi:digitale-gesellschaft.ch' from '!mjbDjyNsRXndKLkHIe:matrix.org'...
Aug 18 13:25:40 matrix-1 synapse[2542335]: synapse.handlers.room: [DELETE-2262217] Failed to leave old room and join new room for '@rettichschnidi:digitale-gesellschaft.ch'
Aug 18 13:25:44 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing events from event_auth_chain_links
Aug 18 13:26:14 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_auth
Aug 18 13:26:21 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_edges
Aug 18 13:26:25 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_json
Aug 18 13:26:48 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_push_actions_staging
Aug 18 13:26:48 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_relations
Aug 18 13:26:48 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_to_state_groups
Aug 18 13:26:48 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_auth_chains
Aug 18 13:26:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_auth_chain_to_calculate
Aug 18 13:26:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from redactions
Aug 18 13:26:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from rejections
Aug 18 13:26:50 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from state_events
Aug 18 13:26:54 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from current_state_events
Aug 18 13:26:55 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from destination_rooms
Aug 18 13:26:55 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_backward_extremities
Aug 18 13:26:55 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_forward_extremities
Aug 18 13:26:55 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_push_actions
Aug 18 13:26:55 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_search
Aug 18 13:26:55 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from partial_state_events
Aug 18 13:26:55 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from events
Aug 18 13:27:47 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from federation_inbound_events_staging
Aug 18 13:27:47 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from local_current_membership
Aug 18 13:27:47 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from partial_state_rooms_servers
Aug 18 13:27:47 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from partial_state_rooms
Aug 18 13:27:47 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from receipts_graph
Aug 18 13:27:47 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from receipts_linearized
Aug 18 13:27:47 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_aliases
Aug 18 13:27:47 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_depth
Aug 18 13:27:47 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_memberships
Aug 18 13:27:48 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_stats_state
Aug 18 13:27:48 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_stats_current
Aug 18 13:27:48 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_stats_earliest_token
Aug 18 13:27:48 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from stream_ordering_to_exterm
Aug 18 13:27:48 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from users_in_public_rooms
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from users_who_share_p
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from appservice_room_l
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from e2e_room_keys
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_push_summar
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from pusher_throttle
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_account_data
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_tags
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from rooms
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] done
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing events from event_auth_chain_links
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_auth
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_edges
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_json
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_push_action
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_relations
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_to_state_gr
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_auth_chains
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_auth_chain_
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from redactions
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from rejections
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from state_events
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from current_state_eve
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from destination_rooms
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_backward_ex
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_forward_ext
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_push_action
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_search
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from partial_state_eve
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from events
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from federation_inboun
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from local_current_mem
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from partial_state_roo
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from partial_state_roo
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from receipts_graph
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from receipts_lineariz
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_aliases
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_depth
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_memberships
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_stats_state
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_stats_curren
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_stats_earlie
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from stream_ordering_t
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from users_in_public_r
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from users_who_share_p
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from appservice_room_l
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from e2e_room_keys
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from event_push_summar
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from pusher_throttle
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_account_data
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from room_tags
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from rooms
Aug 18 13:27:49 matrix-1 synapse[2542335]: synapse.storage.databases.main.purge_events: [DELETE-2262217] [purge] done
Aug 18 13:27:50 matrix-1 synapse[2542335]: synapse.storage.databases.state.store: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from state_groups_state
Aug 18 13:27:50 matrix-1 synapse[2542335]: synapse.storage.databases.state.store: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from state_group_edges
Aug 18 13:27:50 matrix-1 synapse[2542335]: synapse.storage.databases.state.store: [DELETE-2262217] [purge] removing !mjbDjyNsRXndKLkHIe:matrix.org from state_groups
Aug 18 13:27:50 matrix-1 synapse[2542335]: synapse.access.http.8008: [DELETE-2262217] 127.0.0.1 - 8008 - {@admin:digitale-gesellschaft.ch} Processed request: 130.025sec/0.003s

Restart Synapse
Number of entries after deletion:

synapse=# SELECT count(s.room_id) FROM state_groups_state s WHERE room_id='!mjbDjyNsRXndKLkHIe:matrix.org';
 count
--------
 784599
(1 row)

The room does no longer exist on my home server (according to /_synapse/admin/v1/rooms), but almost all entries remain in the state_groups_state table.

Any ideas on how I could help with this issue and/or how I can shrink my database?

squahtx · 2022-08-18T14:22:57Z

There's a known bug where entries remain in state_groups_state after a purge: #12821

DMRobertson added the X-Needs-Info This issue is blocked awaiting information from the reporter label Aug 15, 2022

richvdh closed this as completed Aug 15, 2022

richvdh mentioned this issue Aug 15, 2022

Deleting events causes database corruption #13476

Closed

MadLittleMods added the A-Corruption Things that have led to unexpected state in Synapse or the database label Jun 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERROR - POST-758856 - Failed handle request via 'RoomMembershipRestServlet' – when trying to leave room #13524

ERROR - POST-758856 - Failed handle request via 'RoomMembershipRestServlet' – when trying to leave room #13524

schildbach commented Aug 15, 2022 •

edited by MadLittleMods

Loading

DMRobertson commented Aug 15, 2022

schildbach commented Aug 15, 2022

DMRobertson commented Aug 15, 2022

schildbach commented Aug 15, 2022

schildbach commented Aug 15, 2022 •

edited

Loading

schildbach commented Aug 15, 2022 •

edited

Loading

DMRobertson commented Aug 15, 2022 •

edited

Loading

schildbach commented Aug 15, 2022 •

edited

Loading

richvdh commented Aug 15, 2022

schildbach commented Aug 15, 2022

schildbach commented Aug 15, 2022

richvdh commented Aug 15, 2022

schildbach commented Aug 15, 2022

richvdh commented Aug 15, 2022

schildbach commented Aug 15, 2022

schildbach commented Aug 15, 2022

rettichschnidi commented Aug 18, 2022

squahtx commented Aug 18, 2022

ERROR - POST-758856 - Failed handle request via 'RoomMembershipRestServlet' – when trying to leave room #13524

ERROR - POST-758856 - Failed handle request via 'RoomMembershipRestServlet' – when trying to leave room #13524

Comments

schildbach commented Aug 15, 2022 • edited by MadLittleMods Loading

Description

Steps to reproduce

Homeserver

Synapse Version

Installation Method

Platform

Relevant log output

Anything else that would be useful to know?

DMRobertson commented Aug 15, 2022

schildbach commented Aug 15, 2022

DMRobertson commented Aug 15, 2022

schildbach commented Aug 15, 2022

schildbach commented Aug 15, 2022 • edited Loading

schildbach commented Aug 15, 2022 • edited Loading

DMRobertson commented Aug 15, 2022 • edited Loading

schildbach commented Aug 15, 2022 • edited Loading

richvdh commented Aug 15, 2022

schildbach commented Aug 15, 2022

schildbach commented Aug 15, 2022

richvdh commented Aug 15, 2022

schildbach commented Aug 15, 2022

richvdh commented Aug 15, 2022

schildbach commented Aug 15, 2022

schildbach commented Aug 15, 2022

rettichschnidi commented Aug 18, 2022

squahtx commented Aug 18, 2022

schildbach commented Aug 15, 2022 •

edited by MadLittleMods

Loading

schildbach commented Aug 15, 2022 •

edited

Loading

schildbach commented Aug 15, 2022 •

edited

Loading

DMRobertson commented Aug 15, 2022 •

edited

Loading

schildbach commented Aug 15, 2022 •

edited

Loading