-
Notifications
You must be signed in to change notification settings - Fork 2
Multi Model DB
Storing data in graph form has many benefits for application design. The main trade-off is the performance cost of normalizing data during writes, and de-normalizing data during queries. Therefore minimizing the number of unnecessary query executions is critical to any graph data architecture.
The Reflet db leverages an important property of queries that conform to EQL (a subvariant of Datomic pull) to implement an event sourced, reactive loop between the db mutation functions and queries. Reflet updates only those reactive queries whose results are affected by a given mutation. Unaffected queries return a cached result.
The result is highly performant, expressive, and simple to use. You can have tens of thousands of reactive queries at once, and still only be limited by the browser's ability to render nodes to the window.
The reflet.db namespace has a more detailed description of the
algorithm in the doc string. Additionally there are lots of practical
examples of the mutation API in the unit
tests,
the Reflet example client, and the debugger implementation itself,
which was written entirely using Reflet.
This document covers:
The db graph mutation functions drive the reactive loop. However they
will only work in the reflet.core event handlers, and not the
re-frame.core ones. Let's see them in action.
Consider a music catalog that consists of possibly nested entity maps:
(require '[reflet.core :as f]
'[reflet.db :as db])
(def catalog
"The UUID strings have been truncated for visual clarity."
[{:system/uuid #uuid "a"
:kr/name "Miles Davis"
:kr.artist/albums #{[:system/uuid #uuid "b"]}}
{:system/uuid #uuid "b"
:kr/name "Miles Smiles"
:kr.album/artist [:system/uuid #uuid "a"]
:kr.album/tracks [{:system/uuid #uuid "t1"
:kr/name "Orbits"
:kr.track/artist {:system/uuid #uuid "c"
:kr/name "Wayne Shorter"}
:kr.track/album [:system/uuid #uuid "b"]
:kr.track/duration 281
:kr/uri "/audio/Miles Smiles - Orbits.mp3"}
{:system/uuid #uuid "t2"
:kr/name "Footprints"
:kr.track/artist [:system/uuid #uuid "c"]
:kr.track/album [:system/uuid #uuid "b"]
:kr.track/duration 281
:kr/uri "/audio/Miles Smiles - Footprints.mp3"}]}])
(f/reg-event-db ::init-catalog
(fn [db _]
(db/mergen db catalog)))The db mutation function db/mergen writes entity transactions to the
immutable db. It normalizes the nested entity maps, unifies by
unique attribute references, and merges them with any matching
entities that already exist in the db. The merge semantics are
clojure.core/merge, not clojure.core/merge-with, which has
important implications for cardinality many attributes. Previous
values are overwritten, not combined, just like you would expect with
regular Clojure maps.
After executing this event handler, the immutable db should look
like:
{::db/data
{[:system/uuid #uuid "a"] {:system/uuid #uuid "a"
:kr/name "Miles Davis"
:kr.artist/albums #{[:system/uuid #uuid "b"]}}
[:system/uuid #uuid "b"] {:system/uuid #uuid "b"
:kr/name "Miles Smiles"
:kr.album/artist [:system/uuid #uuid "a"]
:kr.album/tracks [[:system/uuid #uuid "t1"]
[:system/uuid #uuid "t2"]]}
[:system/uuid #uuid "c"] {:system/uuid #uuid "c"
:kr/name "Wayne Shorter"}
[:system/uuid #uuid "t1"] {:system/uuid #uuid "t1"
:kr/name "Orbits"
:kr.track/artist [:system/uuid #uuid "c"]
:kr.track/album [:system/uuid #uuid "b"]
:kr.track/duration 281
:kr/uri "/audio/Miles Smiles - Orbits.mp3"}
[:system/uuid #uuid "t2"] {:system/uuid #uuid "t2"
:kr/name "Footprints"
:kr.track/artist [:system/uuid #uuid "c"]
:kr.track/album [:system/uuid #uuid "b"]
:kr.track/duration 596
:kr/uri "/audio/Miles Smiles - Footprints.mp3"}}}As you can see, the graph data is just a flat associative
structure. Also, the cardinality and data type of the join
attributes :kr.artist/albums, :kr.album/artist, and
:kr.artist/tracks has been preserved. It is up to you whether you
want to use ordered types or not. Representing ordered graph data
properly across application boundaries can be a pain, but at least
here, isolated in Reflet, it is simple enough.
All of the Reflet db mutation functions are pure, and the db is
just an immutable map, so we can freely interleave regular Clojure
operations with graph ones:
(f/reg-event-db ::init-catalog
(fn [db _]
(-> db
(db/mergen catalog)
(assoc ::flag true))))Both the graph and non-graph data lives in the same db, the same single source of truth. This provides a great deal of flexibility when choosing data models across your program. It also makes bringing graph data models to existing Re-frame applications and event handlers easier to do incrementally.
You can still query for the non-graph ::flag using regular re-frame
subscriptions:
(f/reg-sub ::get-flag
(fn [db _]
(get db ::flag)))In the Graph Queries document we will see how to define queries for the graph data.
Both db/assoc-inn and db/update-inn mostly adhere to the semantics
of their Clojure counterparts. One difference is that the first
element in their path must be a reference.
Let's add the year that the album was released:
(f/reg-event-db ::add-year
(fn [db [_ album-ref year]]
(db/assoc-inn [album-ref :kr.album/year] year)))
(f/disp [::add-year [:system/uuid #uuid "b"] "1967"])
;; Entity becomes...
{:system/uuid #uuid "b"
:kr/name "Miles Smiles"
:kr.album/year "1967"
:kr.album/artist [:system/uuid #uuid "a"]
:kr.album/tracks [[:system/uuid #uuid "t1"]
[:system/uuid #uuid "t2"]]}Let's increment a track play count:
(f/reg-event-db ::inc-play-count
(fn [db [_ track-ref]]
(db/update-inn db [track-ref :kr.track/plays] inc)))
(f/disp [::inc-play-count [:system/uuid #uuid "t2"]])
;; Entity becomes...
{:system/uuid #uuid "t2"
:kr/name "Footprints"
:kr.track/artist [:system/uuid #uuid "c"]
:kr.track/album [:system/uuid #uuid "b"]
:kr.track/plays 1
:kr.track/duration 596,
:kr/uri "/audio/Miles Davis - Footprints.mp3"}Or apply an update function to the entire track entity:
(f/reg-event-db ::remove-play-count
(fn [db [_ track-ref]]
(db/update-inn db [track-ref] dissoc :kr.track/plays)))
(f/disp [::remove-play-count [:system/uuid #uuid "t2"]])
;; Entity becomes...
{:system/uuid #uuid "t2"
:kr/name "Footprints"
:kr.track/artist [:system/uuid #uuid "c"]
:kr.track/album [:system/uuid #uuid "b"]
:kr.track/duration 596,
:kr/uri "/audio/Miles Davis - Footprints.mp3"}This last example is the preferred way to remove an attribute from a graph entity.
Note that the Reflet graph associative operations assoc-inn and
update-inn will not resolve beyond any joins in their paths. So
the following will not work:
(f/reg-event-db ::update-play-count-via-album
(fn [db [_ album-ref]]
(db/update-inn db [album-ref :kr.album/tracks 0 :kr.track/plays] inc)))If we refer back to the graph data in the db:
{::db/data
{[:system/uuid #uuid "b"] {:system/uuid #uuid "b"
:kr/name "Miles Smiles"
:kr.album/artist [:system/uuid #uuid "a"]
:kr.album/tracks [[:system/uuid #uuid "t1"]
[:system/uuid #uuid "t2"]]}
[:system/uuid #uuid "t1"] {:system/uuid #uuid "t1"
:kr/name "Orbits"
...}
...}}We see that the :kr.album/tracks attribute in the album entity is
cardinality many reference. The first of those references (index 0)
is to the track "Orbits". To update :kr.track/plays in the album
entity via that join, the join would need to be positionally resolved,
taking into account the order in the join. But the semantics of this
kind of positional resolution would be complex and error prone in the
general case, and so this is not supported. Both db/assoc-inn and
db/update-inn throw an error if given paths longer than two
elements.
Sometimes instead of a random reference, you just want to store one or
more graph entities at a semantically meaningful key in the
db. Let's say you want to store an application's active user at the
:active/user key. You can do this with db/assocn:
(f/reg-event-db ::set-active-user
(fn [db [_ user]]
(db/assocn db :active/user user)))Here, user could be a reference:
[:system/uuid #uuid "user"]
or an entity map:
{:system/uuid #uuid "user"
:kr/name "Name"
:kr/role "user"}If user is an entity map, db/assocn will first normalize and merge
any new data into the db:
{::db/data
{:active/user [:system/uuid #uuid "user"] ; <- cardinality one join
[:system/uuid #uuid "user"] {:system/uuid #uuid "user" ; <- merged in user
:kr/name "Name"
:kr/role "user"}}}The :active/user entry, which is not a reference, is called a link
entry and points directly to a join.
Link entries can be cardinality one or many, and you can use them in event handlers and queries to access graph data (more on that in the Graph Queries document).
For example, you could add a link entry to a global list of favourite tracks:
(f/reg-event-db ::set-favourite-tracks
(fn [db _]
(->> [{:system/uuid #uuid "t3"
:kr/name "Freedom Jazz Dance"
:kr.track/artist [:system/uuid #uuid "c"]
:kr.track/album [:system/uuid #uuid "b"]}
{:system/uuid #uuid "t4"
:kr/name "Dolores"
:kr.track/artist {:system/uuid #uuid "d"
:kr/name "Eddie Harris"}
:kr.track/album [:system/uuid #uuid "b"]}]
(db/assocn db :favourite/tracks))))Which would produce a db that includes the :favourite/tracks link
entry and the new data:
{::db/data
{...
:favourite/tracks [[:system/uuid #uuid "t3"]
[:system/uuid #uuid "t4"]]
[:system/uuid #uuid "t3"] {:system/uuid #uuid "t3"
:kr/name "Freedom Jazz Dance"
:kr.track/artist [:system/uuid #uuid "c"]
:kr.track/album [:system/uuid #uuid "b"]}
[:system/uuid #uuid "t4"] {:system/uuid #uuid "t4"
:kr/name "Dolores"
:kr.track/artist [:system/uuid #uuid "a"]
:kr.track/album [:system/uuid #uuid "b"]}
[:system/uuid #uuid "d"] {:system/uuid #uuid "d"
:kr/name "Eddie Harris"}}}Once you have asserted a link entry, you can update it using
updaten:
(f/reg-event-db ::add-favourite
(fn [db [_ track-ref]]
(db/updaten db :favourite/tracks conj track-ref)))
(f/disp [::add-favourite [:system/uuid #uuid "t2"]])
;; The link entry becomes
{::db/data
{:favourite/tracks [[:system/uuid #uuid "t3"]
[:system/uuid #uuid "t4"]
[:system/uuid #uuid "t2"]]
...}}Note that unlike assocn, updaten only updates the link entry and
will not do any normalization or merging of the additional arguments
that are passed to the update function. You may need to combine it
with the other db mutation functions, like mergen, depending on the
context.
Both regular reference entries and link entries can be removed from
the multi-model db using db/dissocn:
(f/reg-event-db ::remove-stuff
(fn [db [_ artist-ref]]
(db/dissocn db artist-ref :active/user)))Importantly, db/dissocn does not automatically clean up joins
elsewhere in the db that resolve to removed entities. For example,
after removing the artist in the above example, the :kr.track/artist
ref in each track entity will still point to the removed
artist. Ultimately the correct behaviour in this situation is context
dependent, and so for now it is left to user code decide whether any
cleanup is necessary, and how to do it. It is sufficient to note that
entity references that do not resolve anywhere will not break queries
and joins on them will return nil during de-normalization.
Finally, note that out of all the db mutation functions, only
db/mergen and db/assocn normalize nested entity maps.
The Reflet multi-model db has a concept of time: an internal db tick that monotonically, but not contiguously, increases with each db mutation. This enables two things:
-
Time travel: while the multi-model db algorithm is event sourced, you can cycle to any previous value of the db and all queries should update reactively, whether an event occurred or not. This means that Reflet works well with libraries like
re-frame-10xthat let you cycle through the db history along-side the live app. -
The Reflet debugger uses the db tick,
t, to help you understand the sequence of events, query updates and FSM transitions. See the Debugging document for more.
Just do not rely on t to increase contiguously.
A current limitation of the Reflet DB implementation is that it cannot unify more than one unique attribute reference to the same entity. Specifically, the following will have undefined behaviour:
(f/disp [::f/config {:id-attrs #{:kr.unique/attr1 :kr.unique/attr2}}])
(db/mergen db [{:kr.unique/attr1 #uuid "a"
:kr.unique/attr2 #uuid "b"}])The current implementation has no way to consistently decide which of
the two attributes should be used for references: [:kr.unique/attr1 #uuid "a"] or the [:kr.unique/attr2 #uuid "b"].
There is a proposed implementation to resolve this, but until there appears to be significant need this feature, it will be kept in the backlog.
Until then it is recommended that you choose one canonical unique attribute for each entity. In practice, this limitation has been fairly easy to work around, even for complex applications.
If you need to generate references in your event handlers, do not use
the reflet.core/with-ref macro, reflet.db/random-ref or
cljs.core/random-uuid directly. In order to keep your event handlers
pure, use the :reflet.core/with-ref coeffect:
(f/reg-event-fx ::init-catalog
(f*/inject-cofx ::f/with-ref {:system/uuid [:artist :album]})
(fn [{db :db
{a :artist
b :album} ::f/with-ref} _]
{:db (db/mergen db [{:system/uuid (second a)
:kr/name "Miles Davis"
:kr.artist/albums #{b}}
{:system/uuid (second b)
:kr/name "Miles Smiles"}])}))This has almost the same semantics as the with-ref macro, except
because it is used outside of a reactive context, it will only ever
make persistent references.
Next: Graph Queries
Home: Home