Support UNSPECIFIED payload format in CloudEvents mapping (#248)

* Support UNSPECIFIED payload format in CloudEvents mapping

This is a follow up of
#247

* Change the type name of uProtocol messages

The names defined for the different uProtocol message types have been
changed to allow distinguishing between existing CloudEvents being
exchanged by legacy systems using a predecessor of Eclipse uProtocol
and CloudEvents conforming to the mapping rules defined in the
uProtocol specification.

up-core-api/uprotocol/v1/uattributes.proto

@@ -70,27 +70,32 @@ message UAttributes {
 // uProtocol defines different types of messages.
 // Using the message type, validation can be performed to ensure transport
 // validity of the data in the {@link UAttributes}.
+//
+// The "up-" prefix used in the uprotocol.ce_name options has been introduced
+// to allow distinguishing between existing CloudEvents being exchanged by
+// legacy systems using a predecessor of Eclipse uProtocol and CloudEvents
+// conforming to the mapping rules defined in the Eclipse uProtocol specification.
 enum UMessageType {
     // Unspecified message type
     UMESSAGE_TYPE_UNSPECIFIED = 0;
 
     // Publish
     // A message that is used to notify all interested consumers of an event that has occurred.
-    UMESSAGE_TYPE_PUBLISH = 1 [(uprotocol.ce_name) = "pub.v1"];
+    UMESSAGE_TYPE_PUBLISH = 1 [(uprotocol.ce_name) = "up-pub.v1"];
 
     // RPC Request
     // A message that is used by a service consumer to invoke one of a service provider's methods with some input data, expecting the service
     // provider to reply with a response message.
-    UMESSAGE_TYPE_REQUEST = 2 [(uprotocol.ce_name) = "req.v1"];
+    UMESSAGE_TYPE_REQUEST = 2 [(uprotocol.ce_name) = "up-req.v1"];
 
     // RPC Response
     // A message that is used by a service provider to send the outcome of processing a request message
     // from a servcice consumer.
-    UMESSAGE_TYPE_RESPONSE = 3 [(uprotocol.ce_name) = "res.v1"];
+    UMESSAGE_TYPE_RESPONSE = 3 [(uprotocol.ce_name) = "up-res.v1"];
 
     // Notification
     // A message that is used to inform a specific consumer about an event that has occurred.
-    UMESSAGE_TYPE_NOTIFICATION = 4 [(uprotocol.ce_name) = "not.v1"];
+    UMESSAGE_TYPE_NOTIFICATION = 4 [(uprotocol.ce_name) = "up-not.v1"];
 }
 
 

up-l1/cloudevents.adoc

@@ -47,15 +47,15 @@ The https://cloudevents.io/[CloudEvents] project has defined a set of *Formats*
 
 The accompanying *Bindings* describe how events can be sent via well-known transport protocols like https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/bindings/http-protocol-binding.md[HTTP], https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/bindings/mqtt-protocol-binding.md[MQTT] etc.
 
-[.specitem,oft-sid="dsn~cloudevents-umessage-mapping~1",oft-needs="impl,utest",oft-title="Mapping of UMessages to CloudEvents"]
+[.specitem,oft-sid="dsn~cloudevents-umessage-mapping~2",oft-needs="impl,utest",oft-title="Mapping of UMessages to CloudEvents"]
 == uMessage Mapping
 
 uProtocol uses xref:../basics/umessage.adoc[UMessages] to exchange information between uEntities.
 The following sections define the mapping between UMessages and CloudEvents.
 
 === uAttributes Mapping
 
-The table below defines the mapping of uAttributes to CloudEvent attributes. This mapping *MUST* be used by uProtocol Transport Libraries which use the Protobuf Event Format for the binding to a particular transport protocol.
+The table below defines the mapping of uAttributes to CloudEvent attributes. This mapping *MUST* be used by uProtocol Transport Libraries which use CloudEvents for binding to a particular transport protocol.
 
 .Mapping of uAttributes to CloudEvent properties
 [width="100%",cols="12%,8%,10%,70%",options="header",]
@@ -155,7 +155,7 @@ a|A code indicating an error that has occurred during the delivery of either an
 |`payload_format`
 |`pformat`
 |Integer
-|The value of the UPayloadFormat that is used to indicate the encoding of the payload
+|The value of the UPayloadFormat that is used to indicate the encoding of the payload (if any). The concrete mapping rules are defined in <<ce-formats>>.
 
 |===
 
@@ -164,6 +164,8 @@ a|A code indicating an error that has occurred during the delivery of either an
 
 The sections below define the mapping of uPayload to CloudEvent attributes.
 
+NOTE: The custom `pformat` property is used to indicate the payload type instead of the standard `datacontenttype` property defined by CloudEvents. Its value is set to the integer value assigned to the link:../up-core-api/uprotocol/uattributes.proto[UPayloadFormat]. This helps to reduce the size of the resulting data structure.
+
 ==== Mapping to CloudEvent Protobuf Format
 
 The rules defined in https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/formats/protobuf-format.md[Protobuf Event Format for CloudEvents, Version 1.0.2] MUST be applied when mapping UPayload to CloudEvents using the Protobuf Format. The table below defines specific values to use for the different UMessage payload types.
@@ -172,40 +174,45 @@ This mapping *MUST* be used by uProtocol Transport Libraries which use the Proto
 
 [%autowidth]
 |===
-|UPayload Type |CE `datacontenttype` |CE `dataschema` |CE Property to map Payload Data to
+|UPayloadFormat |CE `pformat` |CE `dataschema` |CE Property to map Payload Data to
 
-|`UPAYLOAD_FORMAT_PROTOBUF_WRAPPED_IN_ANY`
-|`-`
+|`UPAYLOAD_FORMAT_UNSPECIFIED`
 |`-`
+|*MAY* be set to a URI-Reference identifying the schema that the data adheres to
+|`binary_data`
+
+|`UPAYLOAD_FORMAT_PROTOBUF_WRAPPED_IN_ANY`
+|`0x01`
+|*MAY* be set to `type.googleapis.com/google.protobuf.Any`
 |`proto_data`
 
 |`UPAYLOAD_FORMAT_PROTOBUF`
-|`application/protobuf`
-|*SHOULD* be set to a URI-Reference identifying the schema that the data adheres to
+|`0x02`
+|*SHOULD* be set to the protobuf's type URL
 |`proto_data`
 
 |`UPAYLOAD_FORMAT_JSON`
-|`application/json`
+|`0x03`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`text_data`
 
 |`UPAYLOAD_FORMAT_SOMEIP`
-|`application/x-someip`
+|`0x04`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`binary_data`
 
 |`UPAYLOAD_FORMAT_SOMEIP_TLV`
-|`application/x-someip_tlv`
+|`0x05`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`binary_data`
 
 |`UPAYLOAD_FORMAT_RAW`
-|`application/octet-stream`
+|`0x06`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`binary_data`
 
 |`UPAYLOAD_FORMAT_TEXT`
-|`text/plain`
+|`0x07`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`text_data`
 
@@ -219,40 +226,45 @@ This mapping *MUST* be used by uProtocol Transport Libraries which use the JSON
 
 [%autowidth]
 |===
-|UPayloadFormat |CE `datacontenttype` |CE `dataschema` |CE Property to map Payload to
+|UPayloadFormat |CE `pformat` |CE `dataschema` |CE Property to map Payload to
 
-|`UPAYLOAD_FORMAT_PROTOBUF_WRAPPED_IN_ANY`
-|`-`
+|`UPAYLOAD_FORMAT_UNSPECIFIED`
 |`-`
+|*MAY* be set to a URI-Reference identifying the schema that the data adheres to
+|`data_base64`
+
+|`UPAYLOAD_FORMAT_PROTOBUF_WRAPPED_IN_ANY`
+|`0x01`
+|*MAY* be set to `type.googleapis.com/google.protobuf.Any`
 |`data_base64`
 
 |`UPAYLOAD_FORMAT_PROTOBUF`
-|`application/protobuf`
-|*SHOULD* be set to a URI-Reference identifying the schema that the data adheres to
+|`0x02`
+|*SHOULD* be set to the protobuf's type URL
 |`data_base64`
 
 |`UPAYLOAD_FORMAT_JSON`
-|`application/json`
+|`0x03`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`data`
 
 |`UPAYLOAD_FORMAT_SOMEIP`
-|`application/x-someip`
+|`0x04`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`data_base64`
 
 |`UPAYLOAD_FORMAT_SOMEIP_TLV`
-|`application/x-someip_tlv`
+|`0x05`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`data_base64`
 
 |`UPAYLOAD_FORMAT_RAW`
-|`application/octet-stream`
+|`0x06`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`data_base64`
 
 |`UPAYLOAD_FORMAT_TEXT`
-|`text/plain`
+|`0x07`
 |*MAY* be set to a URI-Reference identifying the schema that the data adheres to
 |`data`
 
@@ -267,30 +279,30 @@ CloudEvents *SHOULD* only be serialized when they are about to be sent via the x
 The following examples are using the CloudEvent JSON Format.
 
 === Publish
-[source]
+[source, json]
 ----
 {
     "specversion": "1.0",
     "id": "cf8b1bcd-30bd-43be-a8d3-ad1cde652e10",
     "source": "//VCU.VIN/body.access/1/door.front_left#Door",
-    "type": "pub.v1",
+    "type": "up-pub.v1",
     "priority": "CS1",
     "ttl": 10000,
-    "datacontenttype": "text/plain",
+    "pformat": 7,
     "data": "open"
 }
 ----
 
 === Notification
-[source]
+[source, json]
 ----
 {
     "specversion": "1.0",
     "id": "cf8b1bcd-30bd-43be-a8d3-ad1cde652e10",
     "source": "//VCU.VIN/body.access/1/door.front_left#Door",
     "sink": "//VCU.VIN/companion.app/1/status.update",
-    "type": "not.v1",
-    "datacontenttype": "application/json",
+    "type": "up-not.v1",
+    "pformat": 3,
     "data": {
         "subject": "door.front_left",
         "status": "open"
@@ -299,34 +311,35 @@ The following examples are using the CloudEvent JSON Format.
 ----
 
 === Request
-[source]
+[source, json]
 ----
 {
     "specversion": "1.0",
     "id": "cf8b1bcd-30bd-43be-a8d3-ad1cde652e10",
     "source": "//VCU.VIN/MyApp/1/rpc.response",
     "sink": "//VCU.VIN/body.access/1/rpc.UpdateDoor",
-    "type": "req.v1",
+    "type": "up-req.v1",
     "priority": "CS4",
     "ttl": 50000,
+    "pformat": 1,
     "data_base64": "... base64 encoded serialization of UpdateDoorRequest packed
                         in google.protobuf.Any ..."
 }
 ----
 
 === Response
-[source]
+[source, json]
 ----
 {
     "specversion": "1.0",
     "id": "5b9fe861-8c1c-4899-9b07-ad1cde652e10",
     "source": "//VCU.VIN/body.access/1/rpc.UpdateDoor",
     "sink": "//VCU.VIN/MyApp/1/rpc.response",
-    "type": "res.v1",
+    "type": "up-res.v1",
     "priority": "CS4",
     "reqid": "cf8b1bcd-30bd-43be-a8d3-ad1cde652e10",
     "ttl": 50000,
-    "datacontenttype": "application/protobuf",
+    "pformat": 2,
     "dataschema": "type.googleapis.com/google.rpc.Status",
     "data_base64": "... base64 encoded serialization of google.rpc.Status ..."
 }

up-l2/dispatchers/README.adoc

@@ -83,28 +83,28 @@ When a dispatcher is unable to dispatch an event for a given reason (queue full,
 |*google.rpc.Code* |*Reason*
 
 |`*UNAVAILABLE*`
-|The req.v1 has expired due to the downstream uE was unavailable (ex. uDevice was disconnected). uE that issued the req.v1 MAY retry with back-off
+|The service is not available (at the moment), e.g. because the device that the service is deployed to is being restarted. The client MAY retry the request but SHOULD use an exponential back-off for doing so.
 
 |`*DEADLINE_EXCEEDED*`
-|CE has timed out per the ttl attribute specifications defined in req.v1 event
+|The time-to-live set in the client's request has expired.
 
 |`*PERMISSION_DENIED*`
-|source is not permitted to access sink
+|The client is not authorized to invoke the service operation.
 
 |`*UNAUTHENTICATED*`
-|source does not have valid authentication credentials (ex. uE's identity does not match the source attribute)
+|The credentials provided by the client could not be verified, e.g. because the client's identity does not match the source attribute.
 
 |`*RESOURCE_EXHAUSTED*`
-|The dispatcher ran out of resources (buffer full)
+|The dispatcher ran out of resources, e.g. because a message queue's maximum capacity has been reached.
 
 |`*INVALID_ARGUMENT*`
-|Invalid CE header attributes not covered above (ex. any mal-formatted attributes)
+|The service can not process the request message, e.g. because an attribute or the payload does not have the expected format.
 
 |`*UNKNOWN*`
-|An unknown (but not critical) error has occurred
+|An unknown (but not critical) error has occurred.
 
 |`*INTERNAL*`
-|There is a serious error has occurred not described by error codes mentioned above
+|A serious error has occurred that is not covered by any of the error codes mentioned above.
 !===
 
 <<rpc-error-flow>> figure below illustrates the sequence of messages for RPC flows and the role dispatchers play in error handling.

up-l3/README.adoc

@@ -33,10 +33,10 @@ uProtocol supports the architecture patterns defined in the table below to imple
 
 |*RPC* |At-least-once |When uE requires acknowledgement from the receiver  a|
 * If the client-side business logic requires retry policy, it *MUST* implement the retry policy
-* Server-side business logic *MUST* implement Idempotency for remote procedural calls
-* Dispatchers that are unable to deliver req.v1, *MUST* generate a _delivery failure res.v1_ and send it back to the Sender uE
+* Server-side business logic *MUST* implement Idempotency for service operations.
+* Dispatchers that are unable to deliver an RPC request message, *MUST* generate a _delivery failure_ and return it back to the client uE
 
-|*Publication* |At-most-once |When uE wishes to publish an CE to multiple consumers (a.k.a. "fire & forget") a|
+|*Publication* |At-most-once |When uE wishes to publish an CE to multiple consumers (a.k.a. _fire & forget_) a|
 * Dispatcher *MUST* provide access to CEs that have failed to be delivered (DLT defined in a later section), for both the Sender and/or Receiver uE business logic
 
 |*Notification* |At-most-once |When a uE wishes to notify a specific uE (a.k.a a publication with a destination) |