Public/Private events

[unixslayer]

In the topic of integration between models, there is a need to distinguish between private and public events.

Private events should be understood as those that can be stored as state changes but not necessarily published. Using them for integration in a distributed architecture reveals implementation details that are part of an independent model.

Public events are part of the so-called published language and do not have to contain information about the state change, but only inform about some specific situation that occurred in the system during the handling of the command, such as a summary of the operation.

I propose to introduce the possibility to specify which events registered (e.g. by the aggregate) will be public and which private with the following assumptions:

• private events will be recorded registered by the ES aggregate will be recorded on the stream but will not be published by sending them to EventBus
• public events will not be saved but only published by sending them to the EventBus

Example:

#[PrivateEvent]
final readonly class StateWasChanged
{
    public function __construct(
        public string $aggregateId
        public string $previousState,
        public string $newState,
        public string $actor,
        public string $internalAttribute,
    ) {
    }
}

#[PublicEvent]
final readonly class AggregateWasChanged
{
    public function __construct(
        public string $aggregateId
    ) {
    }
}

To ensure backward compatibility, events that are not specified as private nor public will not change the current behavior of AggregateFlow - they will be both saved and published.

The idea directly affects the Projections that handle published private events. The solution may be to configure an internal EventBus to allow private events to be sent only to channels defined on the projections.

[dgafka]

Hey,

My convention on Public vs Private is slight different:

• Public Events are Events which are cross Service Events, therefore the coupling is introduced between different Contexts, and versioning is most likely needed to avoid B/C.
• Private Events on other hand are part of single Service Level (most likely mapped to same Bounded Context) and are part of single deployment unit. Therefore changes to the structure of this Event and to the handling process of this Event goes live together. Therefore there is no need to do versioning, as this is all part of single boundary, single deployment unit.

So the Event Bus (Event Handler) was meant to be used on within single Service level, not to be used to integrate across boundary where the need for decoupling and possibility of doing B/C exists.
For those situations integration was meant to be done via Distributed Bus (Distributed Event Handler), as then we can filter out what goes as public, transform those Events to decouple their structure, or combine different events into single public one.
I was describing this process in more details in one of the blog posts.

So that being said, what about your use case?
Maybe the integration can be change to not depend on the Event Bus?

[unixslayer]

Seems to me we have the same understanding of Public and Private Events. Although in my case I have a modular monolith in which Bounded Contexts are fairly defined and each BC is treated as a single deployment unit on code level yet still deployed as one. That makes technical aspect to impact strategic design in a way I think may be to avoid.

I will have to check, but can I be using Distributed Bus with monolitic architecture? I guess my problem here is using Event Bus for both public and private events due there is no actual difference between them at the moment unless we change architecture.

Anyway, currently events publishing using Event Bus is one of the steps of aggregate flow. If Ecotone provide a possibility to declare which events are public and which are private it may create additional configuration for that flow which may put more focus on strategic design.

What do you think?

[dgafka]

Distributed Bus with current implementation could be used, if those would be loaded as separate Symfony Applications, which I believe will not be a case, right?

---

I had similar cases in past where I was having multiple contexts within same Service (deployed together). But I still found the need in some scenarios to communicate between them, this way or another. For example Invoice was generated in Billing Context, and Payment Context triggered an automatic payment based on this.
I would say then, it make sense then to have it published, but published within given Bounded Context, so it does not cross the boundary.
But then we need to have a way to integrate those Contexts in more explicit way.
Maybe we should allow to define those bounded context per application namespace and build some customizated version of Distributed Bus?

e.g.

namespace App

public function configuration(): LocalDistributedBus()
{
      return LocalDistributedBus(
           [
                'billing_service' => 'App\Billing'
                'payment_service' => 'App\Payment'
           ]
     )
}

Billing Context

   namespace App\Billing;

    public function onInvoiceGenerated(
          InvoiceGenerated $event, 
          #[Reference("localDistributedBus")] DistributedBus $distributedBus): void
    {
          $distributedBus->sendCommand(
               'payment_service',
               'payment_context.make_payment',
               //use array, object however you want to decouple
               [''customerId' => $event->customerId, 'invoice' => $event->invoiceId, 'amount' => $event->totalAmount)]
           );
    }

Payment Context

   namespace App\Payment;

   #[Distributed]
   #[CommandHandler('payment_context.make_payment')]
    public function makePayment(MakePayment $command): void
    {
          // do the payment like a Boss
    }   

And then what we could do is, to disallow to bind to Events from not your own Context (I guess by namespace?).
Wdyt?

[lifinsky]

Hi! It would be great to have multiple distributed bus instances with different AMQP exchanges or connections (different RabbitMQ instances).

[unixslayer]

Shouldn't Distributed Bus integrate BCs? IMO it should be Event Bus which is considered as local and associated with specific BC. Events which are considered as private should be associated only with one BC therefore, its Event Bus that should be used for broadcasting those locally. Events which are considered as public, can be broadcasted on Distributed Bus to integrate wich other BCs.

Within aggregate flow, which is associated to a single BC, all events should be considered as private by default. Those should be published only in local BC using local Event Bus and, in case of ES Aggregates, persisted on Event Stream.

However, one may want to describe some of those events as public, expecting them to be published on distributed bus once transaction is successfully closed. That would avoid having to call distributed bus directly. For those events it would be expected that they won't be persisted in case of ES Aggregates.

On the other hand, I can imagine having particular events to be consudered both private and public therefore expectation for those would be to be published both in local BC and to Distributed Bus. Also in case of ES Aggregates those should be persisted.

To keep backward compatibility, default configuration should provide single and default Event Bus and/or single Distributed Bus.

[lifinsky]

Multiple distributed event buses are not a popular case, but are real. For example, https://masstransit.io/documentation/configuration/multibus

[dgafka]

Shouldn't Distributed Bus integrate BCs? IMO it should be Event Bus which is considered as local and associated with specific BC.

@unixslayer Ye, I agree. By Local Distributed Bus, I was more of a thinking some like Synchronous Distributed Bus, which could also be useful in testing scenarios (Currently, we do have only AMQP Integration).

Hi! It would be great to have multiple distributed bus instances with different AMQP exchanges or connections (different RabbitMQ instances).

@lifinsky I do see that someone could use both Synchronous and Amqp integration. But what would be the use case to actually use multiple Amqp Distributed Buses?

Within aggregate flow, which is associated to a single BC, all events should be considered as private by default. Those should be published only in local BC using local Event Bus and, in case of ES Aggregates, persisted on Event Stream.

Yep, we could make Events considered as private by default. As by default we would only have one Bounded Context for single Application, therefore no difference to actual behaviour.
However instead of using naming Public/Private, we could follow on existing naming. We do have Distributed Event Handlers, therefore events would be Distributed Events. This also gives a sense that this even goes out, therefore changes to it may cause breaks.

With that we would have two ways to distribute events.

Direct Distribution

#[Distributed]
#[NamedEvent('order.was_placed')]
class OrderWasPlaced
{
     // some data
}

This would make Ecotone automatically distribute the Event use default DistributedBus.
This of course come with downside of sharing an Event between contexts, but is really to use. Just mark an Event with attribute and it's going to be distributed out.

Indirect Distribution

class OrderWasPlaced
{
     // some data
}

class DistributeHandler
{
     // we subscribe to Event in our Bounded Context, and we distribute it the way we want
     #[EventHandler]
     public function handle(OrderWasPlaced $event, DistributedBus $distributedBus)
     {
           $distributedBus->distribute('order.was_placed', [
                   'order_id' => $event->id,
                   // we can add some other data, or skip some to not create coupling
            ]);
 )
     }
}

[dgafka]

Make sense?

[lifinsky]

Public / private is a strange name for microservice / SOA architecture, and also is not part of DDD.

What's really missing:

• have an outbox pattern for both internal asynchronous events and external ones.
• be able to subscribe to distributed events from a separate message bus (separate connection)
• have priority for processing synchronous events

[dgafka]

Public / private is a strange name for microservice / SOA architecture, and also is not part of DDD.

I am not sure where I heard it first, but it's rather used term. Maybe not so often in PHP, as most in-house / opensource implementations that I've seen were actually putting an equal sign between Domain (Private) Events and Integration (Public) Events. Or did not support this kind of level of abstraction at all. :)

[lifinsky]

Domain / Application, Distributed / External. I research in Axon, ABP frameworks.

[unixslayer]

@dgafka kinda makes sense however, I think that attributes like #[NotPersisted] may expose internals too much.

I want this to be as simple as possible. Ecotone provides indirect support for private/public events assuming microservice architecture. Recorded events are by default private and

• persisted and published locally for ES Aggregates
• published locally without being persisted for State Aggregates and direct usage of Event Bus

In order of exposing events outside of BC, Indirect Distribution (as you've called it) must be provided. Events sent using Distributed Bus are not persisted. If consumer wants to save them, it is doable but it is not a concern of BC that sent that event.

What I want to achieve is to provide similar support for monolitic architecture and potentially remove Direct Distribution.

I did not consider having private events and not to persist them. This case is already covered by State Aggregates. I guess it may be usefull for ES Aggregates also but I would rather suggets additional configuration for #[PrivateEvent] attribute, eg. skipPersistance: bool.

#[Attribute(Attribute::TARGET_CLASS)]
class PrivateEvent
{
    public function __construct(public readonly bool $skipPersistance = false) {}
}

#[Attribute(Attribute::TARGET_CLASS)]
class PublicEvent
{
}

This won't expose too much of internal logic providing better understanding of modelled BCs. Following examples you've made

// Private Event (published only within boundary, stored)
#[NamedEvent('order.was_placed')
class OrderWasPlaced

// Private Event (published only within boundary, stored) same example but more pure
#[PrivateEvent]
#[NamedEvent('order.was_placed')
class OrderWasPlaced

// Private Event (published only within boundary, not stored)
#[PrivateEvent(skipPersistance: true)]
#[NamedEvent('order.was_placed')
class OrderWasPlaced

// Public Event (published within and outside of the boundary, not stored)
#[PublicEvent]
#[NamedEvent('order.was_placed')
class OrderWasPlaced

For events that may have to be persisted and published both within and outside boudary those looks like to be both private and public. Although, this should be considered rather as a bad practice due being private, those events may contain data sensitive for boudary. Having that I would rather consider following example as hidden feature and not putting too much emphasis for supporting that

// both Private and Public Event (published within and outside of the boundary, persisted)
#[PublicEvent]
#[PrivateEvent]
#[NamedEvent('order.was_placed')
class OrderWasPlaced

This won't break anything in current Ecotone usage due default behavior is preserved. However, it will bring up possibility to improve domain modeling and getting rid of Indirect Distribution.

As for the naming, I don't think we should put too much emphasis here due there is clear definition already described in literature, both Blue Book and Red Book. Bounded Contexts communicate with "outside" world using Published Language, e.g. with Open-Host relations. Therefore, every information Bounded Context will be exposing outside is a published contract and everything else which can't be exposed is private for that BC.

Keep in mind that this discussion, as well as its outcome is about modeling and strategic design, not technical implementation.

[dgafka]

We can map BC on AMQP connection and exchange

@lifinsky you can register multiple publishers with current Ecotone configuration. However there will be always one consumer per Service.
I guess in that case, we would need multiple publisher and multiple consumers per Service, right?

[lifinsky]

@dgafka Do you have tests for this hidden feature?)

[lifinsky]

And what about multiple connections for subscriber (consumer)?

[dgafka]

@dgafka Do you have tests for this hidden feature?)

It's not hidden feature, this is just by design allowing for multiple implementations :)
However I don't think I've been mentioning that anywhere in the docs. You may test it out, but I believe it should work, as single configuration follows exactly the same process of registration.

[dgafka]

And what about multiple connections for subscriber (consumer)?

Consumer needs to be deployed with different connection env, and it will connect to the correct Broker.

AMQP_DSN=something-one bin/console ecotone:run distributed_billing
AMQP_DSN=something-two bin/console ecotone:run distributed_billing

However we won't be able to register same Service under multiple different names. This is done like that to avoid logical confusions.

[lifinsky]

@dgafka This is understandable, but it's not very good practice. It’s like replacing a db connection for a replica without having its separate env variable in the configuration