Remote logging can drop log lines

[anoadragon453]

Description

Remote logging is a feature that allows for sending logging to a remote network resource, as opposed to a local file or the console. Relevant Synapse documentation.

A report from @clokep in #synapse-dev.

Remote logging is fairly broken, just an FYI. It can end up dropping log lines during (re)connection and some other situations. This might render structured logging not as useful.

@Half-Shot chimed in to state that a workaround used in the past was to instead log to disk, and then scrape the resulting files.

@clokep said:

It was something with the way the produced was paused and started IIRC:

https://github.com/matrix-org/synapse/blob/d58dd1d1ff283d14ada805c72aa0b704087ac9b4/synapse/logging/_remote.py#L47

Steps to reproduce

Remote logging can be configured in the logging config. See the bottom of https://element-hq.github.io/synapse/v1.99/structured_logging.html#structured-logging.

[clokep]

Probably worth checking the clokep/ branches -- I wrote a test to reproduce the behavior