MSC3911: Linking media to events #3911
Conversation
turt2live
added
proposal
|
|
||
| ### Overview | ||
|
|
||
| After an item of media is uploaded, it must be linked to an event (via |
There was a problem hiding this comment.
For MLS, we may need to reference media within a to-device event, but we might be able to link it to an in-room event. I'll have to look into this more, but flagging this as something that will need to be considered.
There was a problem hiding this comment.
there's also ephemeral-ish things like user profiles which aren't directly related to events, though are duplicated/copied to membership events in most cases.
richvdh
force-pushed
the
rav/proposal/linking_media_to_events
branch
from
464e5a4
to
cc853ef
Compare
| In addition, this could cause duplication of media in the remote media cache, | ||
| if the implementation does not take steps to deduplicate (eg, storing media | ||
| by content hash rather than media id). |
There was a problem hiding this comment.
Can we spec a response header for the media download that provides a suitable hash of the object?
e.g. X-Matrix-Media-Hash: blake2,AAAAAAAAAAAAAAAAAAAAAAAAAAA....
That way, a homeserver (or client!) which is downloading a copy of the media can abort the download if it finds the hash already in its media store. This means that copies of large media do not lead to each server downloading it for each copy.
(Further, for bandwidth-sensitive clients, a HEAD could be used to get the hash before even initiating the download?)
There was a problem hiding this comment.
De-duplication is already possible at an implementation level, and would probably be handled by a different MSC.
There was a problem hiding this comment.
De-duplication is already possible at an implementation level
I think that reivilibre wasn't talking about the way clients/homeservers are deduplicating files after download, but rather before they're downloaded. I agree that former is an implementation detail that doesn't require uniformity across software, and as such doesn't need to be covered in protocol spec. But if the intention here is to avoid the very process of downloading a file multiple times (from a different linked MXC each time), can this be done without protocol declaring one well known way to discover e.g. that hash of underlying file by HEAD request?
I'm specifically thinking of a situation where you've got a user looking at sticker selector and not-so-maliciously spamming them by clicking them 100s of times. Some people do that, it isn't really a bad behavior in some rooms and contexts (slightly distant example would be custom emote spam on Twitch chats). Since the stickers sent in each event are no longer sharing the same MXC, AFAIK client (and homeserver, if sender is remote) will have to download the sticker 100s of times, and thus generate a lot of unnecessary traffic. Only then it will be able to deduplicate file for local storage/cache.
So my (and I assume reivilibre's) concern here is that with the proposal as it is right now, there's no way for client/homeserver to avoid that "traffic duplication", so to speak. Or is there? 😅
There was a problem hiding this comment.
Hm, on the other hand, that choice of hash type used for deduplication before download, might eventually have an effect on future implementations of dedupe after download. For example, iirc currently in Synapse there's a mapping of MXC URI to local filename which is a random string. With this MSC, each server would have to store a mapping from the new generated IDs to the local file (in Synapse's case, again, a random name). If protocol adds a way to map MXC IDs to file hash early, is there a reason for servers to not store files by naming them after their hashes (as received over e.g. federation)? This would save you one more lookup: MXC IDs -> file hash -> randomized local name.
Oh, and I haven't thought how/if this "early dedupe" could realistically work with files attached to E2EE events, and whether it would add any additional metadata leak other than what's caused by this MSC as is.
| 1. A new "media upload" endpoint is defined, `POST | ||
| /_matrix/client/v1/media/upload`. It is based on the existing | ||
| [`/_matrix/media/v3/upload`](https://spec.matrix.org/v1.4/client-server-api/#post_matrixmediav3upload) | ||
| endpoint, but media uploaded this way is not initially viewable (except to | ||
| the user that uploaded it). This is referred to as a "restricted" media item. |
There was a problem hiding this comment.
TODO: extend this to /_matrix/media/v1/create, as added by MSC2246.
| The new media item can be attached to a new event, and generally functions | ||
| in every way the same as uploading a brand new media item. |
There was a problem hiding this comment.
Can the user copy restricted media they don't have access to? (I assume not, but what is the appropriate error code?)
There was a problem hiding this comment.
I'd say no, it's the same as /download and /thumbnail. 403 and M_UNAUTHORIZED.
| removed. Currently, Synapse does not support removing the events (see also | ||
| [synapse#4720](https://github.com/matrix-org/synapse/issues/4720)); but if at |
There was a problem hiding this comment.
Synapse was fixed a few days ago, apparently :)
| It is expected that servers will continue to treat such media as unrestricted | ||
| (at least for local users), but it would be legitimate for them to, for example, | ||
| return a different `mxc:` URI for each requesting user, and only allow each user | ||
| access to the corresponding `mxc:` URI. |
There was a problem hiding this comment.
I'm not sure I see the advantage of a server issuing distinct URIs for previews. Is there a motivation for putting this here?
There was a problem hiding this comment.
I think I was trying to reason about how we might apply access restrictions to media that is part of a URL preview (if we decided that was a sensible thing to do). It may well be that the extra text confuses more than it clarifies.
There was a problem hiding this comment.
leaving url previews as a way to keep "uploading" unrestricted media would leave open the attack vector this MSC is trying to close.
|
|
||
| ## Unstable prefix | ||
|
|
||
| TODO |
There was a problem hiding this comment.
| TODO | |
| While this MSC is not considered stable, implementations should use the following mapped values. | |
| | Stable | Unstable | | |
| |-|-| | |
| | `/_matrix/client/v1/media/upload` | `/_matrix/client/unstable/org.matrix.msc3911/media/upload` | | |
| | `/_matrix/client/v1/media/create` | `/_matrix/client/unstable/org.matrix.msc3911/media/create` | | |
| | `attach_media` | `org.matrix.msc3911.attach_media` | | |
| | `restrictions` | `org.matrix.msc3911.restrictions` (`event_id` and `profile_user_id` are not prefixed) | | |
| | `/_matrix/client/v1/media/copy/:serverName/:mediaId` | `/_matrix/client/unstable/org.matrix.msc3911/media/copy/:serverName/:mediaId` | |
There was a problem hiding this comment.
(this is what MMR will be using)
| If any of the `attach_media` parameters do not correspond to known | ||
| restricted media items, or they refer to restricted media items that have | ||
| already been attached, the server responds with a 400 error with | ||
| `M_INVALID_PARAM`. |
There was a problem hiding this comment.
Should there also be a limitation that one can only attach media that they uploaded?
There was a problem hiding this comment.
erm... good question.
I think the answer is "yes, there should", to guard against the possibility that someone gets hold of the mxc: URI between it being uploaded and the media being attached to an event. (In which case, they could attach it to their own event and get access to the media.) Particularly in the case of /create rather than /upload, in which the mxc: URI would appear in access logs.
| This "copy" api is to be used by clients when forwarding events with media | ||
| attachments. |
There was a problem hiding this comment.
We'd need to use this for m.sticker events too.
There was a problem hiding this comment.
(and custom emoji, when that exists)
| --gc0p4Jq0M2Yt08jU534c0p | ||
| ``` | ||
|
|
||
| 5. New "media copy" API |
There was a problem hiding this comment.
A downside of the media copy API is clients would no longer be able to reliably cache media. For the example of stickers or emoji, popular/busy rooms could have tens of references to the same media object, which causes tens of downloads because the client doesn't know different.
Possible alternatives (for future MSCs, imo) would be:
- Allow some media objects to be reference counted/linked to multiple events.
- Include a hash of the file in the event, so clients can locally realize that the objects are all the same content-wise.
I'm more preferential to the second.
| --gc0p4Jq0M2Yt08jU534c0p | ||
| ``` | ||
|
|
||
| 5. New "media copy" API |
There was a problem hiding this comment.
going down the rabbit hole of custom emoji, there's an interaction with MSC4027 we will have to figure out.
This comment was marked as duplicate.
This comment was marked as duplicate.
|
|
||
| Fixes [synapse#1263](https://github.com/matrix-org/synapse/issues/1263). | ||
|
|
||
| ## Potential issues |
There was a problem hiding this comment.
@AndrewRyanChama says:
How does this interact with m.replace? Will the client be expected to /copy all of the medias or will it be handled automagically?
| #### Redacting events | ||
|
|
||
| Under this proposal, servers can determine which media is referenced by an | ||
| event that is redacted, and add that media to a list to be cleaned up. |
There was a problem hiding this comment.
The problem with this is that if the user who sent the media leaves the room, the homeserver may no longer get the reaction events. That means there will be no mechanism for those events to be removed.
| Content-Type: application/json | ||
|
|
||
| { "restrictions": { | ||
| "event_id": "$Rqnc-F-dvnEYJTyHq_iKxU2bZ1CI92-kuZq3a5lr5Zg" |
There was a problem hiding this comment.
This means that the association between (encrypted) media and event ids will be plainly accessible to the homeserver, and federated servers. I'd imagine that isn't something we actually care about but something that others may complain about.
Given that /copy presumably can't actually decrypt the media, that also means that the homeserver can associate any copied encrypted media as well
|
|
||
| After an item of media is uploaded, it must be linked to an event (via | ||
| parameters to the `/send` api). A given piece of media is only visible | ||
| to a user if the user can see the corresponding event. |
There was a problem hiding this comment.
Since reference counting is required anyways, I feel like it's better if the server does it automatically. I wrote a short doc on that here: #4086
| The `/_matrix/federation/v1/media/download` and `/_matrix/federation/v1/media/thumbnail` | ||
| endpoints specified by [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) | ||
| are extended: the returned json object may have a property `restrictions`. |
There was a problem hiding this comment.
There was a problem hiding this comment.
there are several updates required to this MSC - the priority has been to get 3916 into working order, then the team will move onto this MSC afterwards.
| The existing endpoint is deprecated. Media uploaded via the existing endpoint | ||
| is "unrestricted". | ||
|
|
||
| 2. Attaching media |
There was a problem hiding this comment.
the proposal is called "Linking media to events" and "linking" terminology is used outside of this list item. is this intentional? can we agree on a single term otherwise? "attaching" seems better suited.
|
|
||
| If any of the `attach_media` parameters do not correspond to known | ||
| restricted media items, or they refer to restricted media items that have | ||
| already been attached, the server responds with a 400 error with |
There was a problem hiding this comment.
reading this proposal top down, it is not entirely clear whether "already attached" means a different event or a duplicate query parameter.
| If the media is not attached to either an event or a profile within a reasonable period | ||
| (say, ten minutes), then the server is free to assume that the user has changed their |
There was a problem hiding this comment.
for clarity: the period starts as the media upload has finished, which is particularly relevant with async uploads?
|
|
||
| The new `/download` and `/thumbnail` endpoints added in | ||
| [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) are | ||
| updated the server must check if the requesting user or server has |
There was a problem hiding this comment.
| updated the server must check if the requesting user or server has | |
| updated such that the server must check if the requesting user or server has |
| cache the restrictions list. | ||
|
|
||
| If neither `event_id` nor `profile_user_id` are present, the requesting | ||
| user should assume that an unknown restriction is present, and not allow access |
There was a problem hiding this comment.
| user should assume that an unknown restriction is present, and not allow access | |
| homeserver should assume that an unknown restriction is present, and not allow access |
| * The only protection for media is the obscurity of the URL, and URLs are | ||
| easily leaked (eg accidental sharing, access | ||
| logs). [synapse#2150](https://github.com/matrix-org/synapse/issues/2150) | ||
| * Anybody (including non-matrix users) can cause a homeserver to copy media |
There was a problem hiding this comment.
since authenticated media landed, I think
| * Anybody (including non-matrix users) can cause a homeserver to copy media | |
| * Any locally authenticated user can cause a homeserver to copy media |
| It is expected that servers will continue to treat such media as unrestricted | ||
| (at least for local users), but it would be legitimate for them to, for example, | ||
| return a different `mxc:` URI for each requesting user, and only allow each user | ||
| access to the corresponding `mxc:` URI. |
There was a problem hiding this comment.
leaving url previews as a way to keep "uploading" unrestricted media would leave open the attack vector this MSC is trying to close.
| * special-casing the bridge AS user to permit it to upload media without | ||
| expiry. |
There was a problem hiding this comment.
| * special-casing the bridge AS user to permit it to upload media without | |
| expiry. | |
| * special-casing the bridge AS user to permit it to upload unrestricted media. |
other ideas:
- attaching media to messages retroactively (e.g. the pastebin copy to the original long message)
- tracking media origin room and/or user so AS don't upload entirely unrestricted media
|
|
||
| Fixes [synapse#1263](https://github.com/matrix-org/synapse/issues/1263). | ||
|
|
||
| ## Potential issues |
There was a problem hiding this comment.
please consider that widgets also use media, i.e. #4039
to summarize, linking media is not a big issue for events of a well-known structure referencing them - with regards to widgets, that means that clients can parse the payload and handle it automatically. however especially for widgets it is often attractive to send nonstandard events (let alone ones with some custom encoding), that prohibit this.
there can be use cases (with widgets or without), where multiple media are attached to an event that represents a certain state at a point in time and which is edited collaboratively. however editing would usually involve changing only one of the referenced media items at most, resulting in an immense overhead when having to copy all the other attached media as well when sending an update (example implementation).
Rendered