Reliable messaging using pipes and channels

[hairyhum]

Proposed pipes and channels abstraction allows to isolate internal message delivery to provide certain delivery properties for the outer messaging.

For example it's possible to have unordered messages inside the pipe, while messages sent by the pipe receiver would be ordered:

Also it's possible to ensure (reasonable) at least once delivery with confirmations and retries between pipe sender and receiver even if the internal communication is not reliable.

This allows to push messaging guarantees to the edges of the messaging system and tolerate failures in certain parts of it.

TODO: follow up with basic pipe logic to provide messaging properties.

[hairyhum]

Prototype implemenrations of ordered pipes and channels using elixir can be found here: https://github.com/ockam-network/ockam/tree/develop/implementations/elixir/ockam/ockam/lib/ockam/messaging/ordering

[hairyhum]

Ordering:
End to end ordering can be implemented with either:

• making sure that sender only processes messages after receiver sent them. This can be called sequential processing or receive buffer approach

• making sure receiver sends messages in the same order that sender received them. This can be called indexed processing or send buffer approach

Prototype implementation can be found in https://github.com/ockam-network/ockam/tree/develop/implementations/elixir/ockam/ockam/lib/ockam/messaging/ordering/strict

[hairyhum]

Delivery reliability:
End-to-end delivery can be implemented with confirmations and retries, ensuring messages are sent by the receiver at least once.

De-duplication of received messages can be implemented with the ordering pipes described above.

[hairyhum]

Prototype implementation: #2126

[hairyhum]

Architecture documentation #2216

[hairyhum]

Dealing with backtraceable route requirement when combining pipes

Problem

Pipes which need to combine multiple properties while some of the properties require
backtraceable routes (a route from receiver to sender)

Pipes are not backtracing the routes, we can't wrap pipes which require backtraceable routes
over other pipes.

If we use function composition pipes (when multiple pipe senders are functions in a single worker),
situation is different, since we have just two workers (Sender and Receiver) and they might have backtraceable route between them.

Replying pipe in the inner edge

In this example resend pipe is a replying pipe, other pipes don't send replies.

[index S -> encrypt S -> resend S] <----> [resend R -> encrypt R -> index R]

Sender needs to handle the repoy by sending it to the last sending handler.

Essentially this option works as if there was a separate resent pipe.

Replying pipe in the middle of the pipeline

If we have a pipe with replies in the middle of the pipeline, it gets more complicated.

In this case resend R needs to send a confirm to resend S, but it cannot go back through encrypt R:

[index S -> resend S -> encrypt S] ---> [encrypt R -> resend R -> index R]
              ^                                         |
              |----------------confirm------------------|

Since we deal with two workers, we might just send message as it is from Receiver to Sender, but we need to call inner message handling for resend pipe.

Let's say we implement handle_inner callback we call when getting inner message in the Sender.

This callback should only be called for resend S and not other handlers, because confirm does not apply to them.

In this case also we only have one handler handling inner messages.

Multiple replying pipes

If we assume that index sender also processes confirms, how do we decide which one should process a certain message?

We could add some metadata to the confirm messages to distinguish which sender handler should handle it.

But it's still important that there is only one handler which processes a single reply

[index S -> resend S -> encrypt S] ---> [encrypt R -> resend R -> index R ]
  ^           ^                                         |         |
  |           |-------------------confirm---------------|         |
  |                                                               |
  ----------------------------index_confirm------------------------

Interestingly, it would be harder to implement that sort of "layer skipping" when using worker based pipe wrapping because receiver would have to get a route to sender somehow.

Delivery properties for replies

Another thing is that now our communication is on multiple layers, messages from sender to receiver have different delivery properties then replies from receiver to sender.

This means that each pipe will have to define internal messaging requirements for both forwarding and replies

For example, if a pipe requires at-least-once delivery in forwards and replies, we can't achieve that by adding the resend layer, becaue for replies it's going to be skipped.

If a pipe requires at-least-once delivery only for forwards - it's fine to add a resend there.

Conclusion

Workers combining multiple pipe behaviours may be used to make those behaviours communicate with each other skipping layers.

Since we can handle replies explicitly different from external messages, we can define a logic to chose which
sender handler to call.

It doesn't make much sense to call all pipes handlers when processing replies, because replies don't go through a pipeline of transformations.

In order to chose a handler to handle reply, we can use:

• a single reply handler per pipe combination (explicitly mark one as replying)
• or use metadata to address a specific handler (index of handler in the list?)

We also need to keep in mind that delivery properties for replies would be different.

[hairyhum]

In addition to the previous post about combining pipe behaviours in a single worker:

Another thing which we want to support is timeouts, which in the actor system are usually implemented as messages. This means that if a pipe handler in a stack wants to have a timer set, the timeout message would be received by the combining worker.

This message would need to be handled by this handler alone so we need some sort of the metadata to tell which handler should process the message, or we need to make sure only one handler can ever have timeouts or other internal messages sent to itself.