Using fmodel to build ES/CQRS apps

[cloudcompute]

Hi

I have few questions, kindly answer.

a. Is this a complete solution for building even-sourcing/CQRS applications? Does it support features like replay, snapshots out-of-the-box?

b. Can we use Nats Jetstream, Mongo, Postgres as an event store?

c. Can we use it in production, stable enough?

d. Also pl. tell about the production status of fmodel-ts (written in TypeScript). Does it implement all the features that fmodel offers?

[idugalic]

Hi @cloudcompute ,

a. It is not a complete solution. FModel is a set of libraries that can be used to accelerate the solution on your end. It only proposes generic infrastructure interfaces on the application module, but the implementation of these interfaces is on you/your team. For example. Please, check the demo application(s) for more

b. Mongo and Postgres can be used as an event stores. First, It is important for the storage technology to support concurrency control so your event streams are totally ordered. Second, it is valuable if your event store can stream events to your Views/Query side (think of concurrent consumers on the Query side!). Unfortunately, SQL/Document Databases are not really made for streaming (but it can be done). I am not knowledgeable about Jetstream, but Mongo and Postgress will work.

c. Yes you can. It is already used in production to my knowledge.

d. fmodel-ts is used in production to my knowledge. fmodel-ts (written in TypeScript) is actually ported from this/kotlin project. It should offer the same features. If you recognize that something is missing, please let us know.

We are actively engaged on all GitHub repos and trying to solve all the bugs or issues as they occur, for your information.

We have fmodel libraries as a general template for your event-sourced information system :) , and you are responsible for

• implementing infrastructure with any DB/storage you find fit the best (price, scale, performance-wise)
• implementing your specific core logic: Decider, View, Saga

We plan to implement more application extensions/modules that will actually provide an implementation for the infrastructure. Application extensions will cover:

• SQL as an event store
• Axon Server
• Event Store

Expected in Summer 2023. We will do this iteratively and release an extension as we find it production ready.

If you are in hurry, please contact me via email ivan.dugalic@fraktalio.com I can share more details (SQL, Axon Server).

PS.
Aggregate Snapshotting is a performance booster feature for the EventSourcedAggregate(s). It will enable storing the current state of your aggregate periodically (after a certain number of events or after some time interval, for example). I will try to add that case to the library. It will be a new EventSourcedSnapshotingAggregate. Roughly, it will be a combination of EventSourcingAggregate and StateStoredAggregate enabling you to store Events and Aggregate State (per some retention policy). I am not a big fan of Snapshotting, and I would rather have a shorter aggregate stream removing the need for Snapshots, nevertheless, it is worth including it in the library as an option.

Materialized View can request a 'replay' of Events from your storage. The question is who is maintaining the token/position of the last event being handled by that Materialized View? Is it infra (DB, Axon Server, Event Store, Kafka) that is doing that and you only need to Ack the last position, or your application is maintaining the Tracking Token on the client side? We believe that this topic is related to Event Streaming (advertising events to Views constantly, as they are stored) and not to Event Sourcing (sourcing the Aggregate state per request) actually. Many different technologies can be used to achieve robust event streaming, but they are very specific in their own design and it usually requires a specific implementation to make it simple and performant.

Best,
Ivan

[cloudcompute]

@idugalic Hi. Thank you for explaining everything in detail. Implementation of Decider, View, and Saga won't be easy though.

Recently, I had started exploring the ES/CQRS existing solutions. I am not in a hurry to have one soon.

Everyone talks about Event Store DB but it has just accumulated 4.7K github stars so far despite the fact that they had open-sourced it in 2013. It is written using C# .NET.. which I want to avoid (because of memory & CPU resources it consumes and spin up time).

Strangely, there is no one else who is offering a product like Event Store DB.

Many developers use Kafka for both event sourcing and pub-sub. But I read several posts where they said that Kafka is not a good-fit for event-sourcing. Also, Kafka's administration is complex.

A very good alternative to Kafka is Nats JetStream. I think it can be used as an event source. It takes very less resources than Kafka and relatively very easy to manage. If you have any idea about whether we should use it for event sourcing or not, kindly tell.

Pl. have a look at this ES/CQRS library. It is written in Elixir, used in production, looks like it offers several features which otherwise are very difficult to implement especially, strong and eventual consistency. It might help you improve the FModel libraries.

Regards
Raman

[idugalic]

You can try Axon Server. It is an event store and message router. (PS. I work with AxonIQ ;) )

Please notice the difference between event sourcing and event streaming. IMHO, Kafka is not a good option for Event Sourcing. The question is how to design an optimal topology of topics and partitions to enable your aggregate to source its own state by reading its own (aggregate) Events before you want to publish a new Event, and please consider optimistic locking mechanisms that actually enables total order/sequence of events (per aggregate). On the other hand, Kafa is good for streaming/advertising events to concurrent consumers that can maintain materialized views, per need. My friend wrote a nice blog on the subject.

Thanks for sharing Elixir libs! I will have a look.

We focused on Domain components at first. and how to utilize programming language features (type system, ADT, pattern matching) to depend on the compiler/transpiler as much as possible. This approach is leaning towards the functional paradigm, but it is not excluding the OOP paradigm totally (OOP can be convenient on the application layers to orchestrate the core domain/logic components).

We plan to focus on the Application layer components this year, by adding more options (Snapshotting, Total/Immediate vs Eventual consistency, for example) and by focusing on Observability more.

Thanks for your feedback, much-appreciated @cloudcompute!

[cloudcompute]

@idugalic

Thanks for your reply.

Good to know that you have plans to extend the functionality of FModels.

Clustering is mandatory for high availability. This feature is not open-source in Axon Server Enterprise and one has to pay which means vendor lock-in. My personal opinion is all features should be open source and a company making money from managed/hosted services so that a customer does not worry about vendor lock-in. Also, Axon Server is not that popular.

As far as Kafka is concerned, honestly speaking, I don't like it at all, the main reason is it is very difficult to administer. As you said, several articles/blogs claim the same: it is not a good fit for event sourcing. Confluent has created a lot of hype around it though.. publicised it so much: Google advertisements, rich content on its website.

Alternatively, Pulsar and Nats that support both streaming (like Kafka) and message queues (like RabbitMQ) can be good candidates for ES. Kafka does not support message queue.

In Java world, there is another ES framework called Eventuate Local that looks pretty stable. I think Axon framework is number one followed by Eventaute.

Can we use Axon framework without using Axon server standard? If yes, is it GraalVM compatible. A combination of Quarkus (based on GraalVM) and Axon framework could be great.

[matthewadams]

...and here's the issue for aggregate snapshots: #202 😊

[idugalic]

Thanks, @matthewadams! I will start working on it (snapshotting) next week in a new feature branch.

[matthewadams]

PS.
Aggregate Snapshotting is a performance booster feature for the EventSourcedAggregate(s). It will enable storing the current state of your aggregate periodically (after a certain number of events or after some time interval, for example). I will try to add that case to the library. It will be a new EventSourcedSnapshotingAggregate. Roughly, it will be a combination of EventSourcingAggregate and StateStoredAggregate enabling you to store Events and Aggregate State (per some retention policy).

Hi, I'm new to the party here. Getting ready to migrate from Axon to our own NATS-based ES+CQRS implementation.

I was glad to read the above paragraph ⬆️, as that's exactly what I expected a snapshotting event-sourced aggregate to be: a combination of of EventSourcingAggregate & StateStoredAggregate. (Nit: English rules would dictate that the new interface name should have two t's: EventSourcedSnapshottingAggregate.). Are you working on this in a branch where I can see the work-in-progress?

Also, @cloudcompute I'm convinced that NATS's JetStream can be used as an event store (see https://github.com/bruth/rita for an initial, prototype implementation in Go), and I think it can support snapshots via a JetStream ObjectStore. See https://www.byronruth.com/implementing-an-event-store-on-nats-part-1/ & https://www.byronruth.com/implementing-an-event-store-on-nats-part-2/. @idugalic, FWIW, I would love to see NATS JetStream added as another option beyond SQL, Axon, & EventStoreDB.

[idugalic]

:) The demo apps and the ref guide are ~10 days old

[idugalic]

BTW, it looks like NATS does not currently support atomic multi-message appends / multiple messages are not appended transactionally. This can be limiting and influence your design decisions. For example, you have to make sure that every command handling process will resolve in a single event/message. Besides that, it looks like it can work. Optimistic locking is in place. At-most-once delivery guarantee is there. I simply have not used NATS as an even store and can't say how performant and robust it can be for event-sourcing purposes.

[bruth]

Hi @idugalic. You are correct that multi-message transactional appends are not supported. There is a lot of interest in this feature primarily driven by the event sourcing use case. However, as you note, it is a design decision for yielding multiple events for a single command. Personally, I am interested in advocating for this in some capacity.

That aside, NATS is really well suited in a variety of other aspects being solid basis for applications leveraging event sourcing and (likely) CQRS.. without requiring any additional external dependencies. Stateless messaging, persistent streams (that can be leveraged as individual event stores) as well as key-value and object-store abstractions on top of the stream primitive.

[idugalic]

Thanks @bruth! Appending multiple events/messages transactionally would be a very nice feature.
BTW, I might choose to combine and compose my command handlers/deciders (domain model) and run it/them in the scope of a single transaction (check here). In this case, every command handler might publish only one event, but all of them should be published in a single transaction. Or, I might choose to run these command handlers/deciders individually, in independent transactions.

Thanks again. I would love to create an example with NATS and learn more about it.

[DylanSale]

FYI I believe the current suggestion from the NATS team to get "transactional" publishing is to publish a single batch event to a topic, then have a consumer that reads that and republishes individual events from it to other topics as needed. You forgo order guarantees in the downstream topics in this case because other events may have been emitted there in the meantime. However, you can get the absolute ordering back at the cost of performance by reading the batch events and extracting the relevant events in order.

One has to consider if NATS is the best tool for the job in that case though. I think we will use a MySQL db as the event store then have a task to publish them to NATS one at a time transactionally, or by reading them from the binlog stream using Maxwell's daemon.

(We are also looking into fmodel with NATS feasibility, it seems to be in the air at the moment!)