From d4ff1c7369286ed85933dd22dd3b63b2ddf37fb0 Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Sat, 6 May 2023 12:10:00 -0600 Subject: [PATCH 1/7] Propose method of specifying custom images in reactions Signed-off-by: Sumner Evans --- proposals/4027-custom-images-in-reactions.md | 111 +++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 proposals/4027-custom-images-in-reactions.md diff --git a/proposals/4027-custom-images-in-reactions.md b/proposals/4027-custom-images-in-reactions.md new file mode 100644 index 00000000000..68af556aa50 --- /dev/null +++ b/proposals/4027-custom-images-in-reactions.md @@ -0,0 +1,111 @@ +# MSC4027 Custom Images in Reactions + +One of the most desired features within the Matrix ecosystem is the ability to +react to messages with custom images. This feature is especially requested by +users who come from Slack and Discord where this functionality is one of the +main ways that the culture of a chat rooms develops. + +There is an existing proposal to +[render image data in reactions (MSC3746)](https://github.com/matrix-org/matrix-spec-proposals/pull/3746/), +but it has had little attention recently and has the flaw of not being conducive +to deduplication (either on the client or server). Sorunome proposed a +modification to that MSC to +[use the MXC URI as the key](https://github.com/matrix-org/matrix-spec-proposals/pull/3746/files#r866285147) +which this proposal adopts. + +This proposal is meant to replace +[MSC3746](https://github.com/matrix-org/matrix-spec-proposals/pull/3746/) and is +additionally intended to document the behaviour of existing clients and bridges. + +Like +[MSC3746](https://github.com/matrix-org/matrix-spec-proposals/pull/3746/), this +MSC does not propose a mechanism for providing a list of available images. + +## Proposal + +This proposal suggests two changes to events with the `m.annotation` relation. + +1. If the `key` of an `m.annotation` relation is an MXC URI of an image, clients + should render the referenced image instead of the key text. + + Detecting if the `key` is an MXC URI can be as sophisticated as the client + wants, but this proposal recommends checking if the string starts with + `mxc://`. + +2. When the annotation's key is an MXC URI, a new (optional) `shortcode` key can + be added to the content of the event with a textual name for the image. + + This shortcode should be shown by clients in situations such as hovering over + the annotation, as alt-text, or if the client does not support rendering + images. + +Example custom image reaction event content + +```json +"content": { + "m.relates_to": { + "rel_type": "m.annotation", + "event_id": "$abcdefg", + "key": "mxc://matrix.org/VOczFYqjdGaUKNwkKsTjDwUa" + }, + "shortcode": ":partyparrot:" +} +``` + +## Potential issues + +### Clients rendering the MXC URI as text + +The biggest disadvantage is that clients that do not support rendering custom +reactions will render the MXC URI as text. However, this is already problematic +because many bridges and clients already support this MSC, and users likely +already encounter this. + +### Un-renderable image referenced in the `key` + +The MXC URI could specify an asset that either does not exist, or is not a +renderable image. Clients can opt to render the `shortcode` in these situations, +or some placeholder/error image, or just opt to render the full key. + +## Alternatives + +### Use the shortcode as the `key` + +This is what was proposed by +[MSC3746](https://github.com/matrix-org/matrix-spec-proposals/pull/3746/). The +problem with this is that there could possibly be multiple distinct images with +the same shortcode. Reactions are only deduplicated based on `key`, so clients +and servers would group these distinct reactions together which is undesirable. + +### Put the `shortcode` as a key within `m.relates_to` + +Instead of being at the root of the `content` dictionary, the `shortcode` value +could be included within `m.relates_to`. This is the wrong place to put this +value because `m.relates_to` is meant to only contain information pertaining to +the relationship between events, not information about the reaction event +itself. + +## Security considerations + +### Image is unencrypted + +Reaction events are not encrypted, and so the MXC URI referenced by the key +would have to be an unencrypted image. However, this is probably not a problem +for the following reasons: + +- Custom reactions are most likely not sensitive information. + +- Users are already able to upload unencrypted content into encrypted rooms, so + this does not introduce any leakage that was not previously possible. + +- Clients can add UX to indicate to users that the reaction images are not + encrypted. + +## Unstable prefix + +Until this proposal is merged into the spec, the `shortcode` key should be +prefixed with `com.beeper.msc4027.`. + +An unstable prefix for the `key` in `m.relates_to` is not necessary as the spec +already allows arbitrary data to be used as the `key`. This MSC merely adds +extra meaning to a specific class of key. From cdfe08915209948ade408a3b4a2d2599cf26d359 Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Sun, 11 Jun 2023 18:09:09 -0600 Subject: [PATCH 2/7] Add mention of multiple MXC URIs pointing to the same file Signed-off-by: Sumner Evans --- proposals/4027-custom-images-in-reactions.md | 22 ++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/proposals/4027-custom-images-in-reactions.md b/proposals/4027-custom-images-in-reactions.md index 68af556aa50..e64e46e8dba 100644 --- a/proposals/4027-custom-images-in-reactions.md +++ b/proposals/4027-custom-images-in-reactions.md @@ -67,6 +67,28 @@ The MXC URI could specify an asset that either does not exist, or is not a renderable image. Clients can opt to render the `shortcode` in these situations, or some placeholder/error image, or just opt to render the full key. +### Multiple MXC URIs can refer to identical files + +Consider two distinct MXC URIs which both refer to a file with identical +contents. If the URIs are used as reaction keys on a given message, +deduplication will not happen as the keys are different. This could cause +confusion as the two reactions would be visually identical despite the keys +being different. + +A similar problem exists already in some clients with Unicode emojis because +there is significant inconsistency as to whether skin tone indicators and other +un-renderable characters should be included in the reaction keys generated by +clients. As such, this proposal does not materially worsen the current +situation. + +Clients can opt to compare the file contents of the images, but this should not +be required as it would complicate rendering significantly. + +Future MSCs may provide a way to share a set of custom reaction images available +in a room/space. This will make it so that users can share custom reaction +images reducing the likelihood of two users uploading the same image under two +different MXC URIs and reacting with them. + ## Alternatives ### Use the shortcode as the `key` From c86338ad686d2c4d753b24787c295693d97e4cdb Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Mon, 12 Jun 2023 09:30:57 -0600 Subject: [PATCH 3/7] Update shortcode unstable field name The Discord bridge already uses com.beeper.reaction.shortcode unstable field name. Signed-off-by: Sumner Evans --- proposals/4027-custom-images-in-reactions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/4027-custom-images-in-reactions.md b/proposals/4027-custom-images-in-reactions.md index e64e46e8dba..fdeeca3cdb8 100644 --- a/proposals/4027-custom-images-in-reactions.md +++ b/proposals/4027-custom-images-in-reactions.md @@ -125,8 +125,8 @@ for the following reasons: ## Unstable prefix -Until this proposal is merged into the spec, the `shortcode` key should be -prefixed with `com.beeper.msc4027.`. +Until this proposal is merged into the spec, the `shortcode` unstable field name +should be `com.beeper.reaction.shortcode`. An unstable prefix for the `key` in `m.relates_to` is not necessary as the spec already allows arbitrary data to be used as the `key`. This MSC merely adds From 75473cc308c001cfee978bea2f6966d06619874d Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Fri, 30 Jun 2023 11:34:58 -0600 Subject: [PATCH 4/7] Remove recommendation for detecting if a string is an MXC URI Signed-off-by: Sumner Evans --- proposals/4027-custom-images-in-reactions.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/proposals/4027-custom-images-in-reactions.md b/proposals/4027-custom-images-in-reactions.md index fdeeca3cdb8..91c3475235f 100644 --- a/proposals/4027-custom-images-in-reactions.md +++ b/proposals/4027-custom-images-in-reactions.md @@ -28,10 +28,6 @@ This proposal suggests two changes to events with the `m.annotation` relation. 1. If the `key` of an `m.annotation` relation is an MXC URI of an image, clients should render the referenced image instead of the key text. - Detecting if the `key` is an MXC URI can be as sophisticated as the client - wants, but this proposal recommends checking if the string starts with - `mxc://`. - 2. When the annotation's key is an MXC URI, a new (optional) `shortcode` key can be added to the content of the event with a textual name for the image. From 5424ea803809d88cdd40585c5cff7ee34534b46c Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Fri, 30 Jun 2023 12:05:15 -0600 Subject: [PATCH 5/7] Specify that shortcode should start & end with ':' Signed-off-by: Sumner Evans --- proposals/4027-custom-images-in-reactions.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/4027-custom-images-in-reactions.md b/proposals/4027-custom-images-in-reactions.md index 91c3475235f..286658c9a79 100644 --- a/proposals/4027-custom-images-in-reactions.md +++ b/proposals/4027-custom-images-in-reactions.md @@ -35,6 +35,8 @@ This proposal suggests two changes to events with the `m.annotation` relation. the annotation, as alt-text, or if the client does not support rendering images. + The shortcode should start and end with the `:` (colon) character. + Example custom image reaction event content ```json From 2ac53284d3b53c61cc7978bd5bd39cac782886f7 Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Fri, 30 Jun 2023 16:31:33 -0600 Subject: [PATCH 6/7] Clarify that shortcode must be a string Signed-off-by: Sumner Evans --- proposals/4027-custom-images-in-reactions.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/4027-custom-images-in-reactions.md b/proposals/4027-custom-images-in-reactions.md index 286658c9a79..0957b795ae9 100644 --- a/proposals/4027-custom-images-in-reactions.md +++ b/proposals/4027-custom-images-in-reactions.md @@ -29,14 +29,14 @@ This proposal suggests two changes to events with the `m.annotation` relation. should render the referenced image instead of the key text. 2. When the annotation's key is an MXC URI, a new (optional) `shortcode` key can - be added to the content of the event with a textual name for the image. + be added to the content of the event with a textual name for the image. This + field must be a string and should start and end with the `:` (colon) + character. This shortcode should be shown by clients in situations such as hovering over the annotation, as alt-text, or if the client does not support rendering images. - The shortcode should start and end with the `:` (colon) character. - Example custom image reaction event content ```json From badc1baa85acd3beffa2463cfee104182f12432c Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Tue, 5 Sep 2023 12:09:44 -0600 Subject: [PATCH 7/7] Add shortcode length check verbiage Signed-off-by: Sumner Evans --- proposals/4027-custom-images-in-reactions.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/proposals/4027-custom-images-in-reactions.md b/proposals/4027-custom-images-in-reactions.md index 0957b795ae9..79983085fd2 100644 --- a/proposals/4027-custom-images-in-reactions.md +++ b/proposals/4027-custom-images-in-reactions.md @@ -37,6 +37,13 @@ This proposal suggests two changes to events with the `m.annotation` relation. the annotation, as alt-text, or if the client does not support rendering images. + The `shortcode` must have a length of less than or equal to 100 bytes + (including the colons at the beginning and end of the string). This + restriction must be enforced by servers when sending reactions, but servers + should not reject events coming across federation due to having too many + bytes in the `shortcode` field. Servers may still opt to locally redact + events having too many bytes in the `shortcode` field. + Example custom image reaction event content ```json