Иммутабельные БД общего назначения
16 November 2022
Факт №1
Datomic (и XTDB) про Факты
“И это есть Факт, Мусьё Дюк!”
(c) Господин Смирнов. ”Неуловимые Мстители. Корона Российской Империи”
Alexander Petrov (a.k.a Lysenko)
Pragmatic Programmer | Sing, Play Sax & Guitar | #Emacs #Clojure #Datomic #DDD #TDD #CleanArchitecture
Старший разработчик Clojure Центра продуктов Дозор в Ростелеком Солар
2015 - 2022 Старший разработчик Ruby / Rails / Erlang в FunBox
2006 - 2014 Enterprise Java Developer / Trainer / Architect / Team Leader / Lecturer
What if we represent part of the world being digitized as a set of situations?
Situation is a set of facts that are consistent with each other as of particular point in time.
We won’t care about anything besides these situations.
Having a log of situations, we might replay the world with a tail recursion in a completely functional fasion.
Source talk: Joe Armstrong Interviews Alan Kay
The Idea of simulating Time along with Process and States appeared in the paper:
1963 Jonh McCarthy Situations, Actions and Causal Laws
2012 Rich Hickey implemented the idea in Datomic database
2012 Stuart Halloway implemented Simulant library to simulation test Datomic based systems
2018-03 JUXT compny created XTDB bitemporal open source EDN database
As of 1979 Great Guitar Player Robben Ford plays in Yellowjackets
| Entity | Attribute | Value | Timestamp |
|---|---|---|---|
| Robben Ford | Plays in | ”Yellowjackets” | 1979 |
Then as of 1986 he plays in Miles Davis’s Band
| Entity | Attribute | Value | Timestamp |
|---|---|---|---|
| Robben Ford | Plays in | ”Miles Davis’s Band” | 1986 |
Imagine General Ledger
We save records on paper
When situation changes we create new records
We NEVER remove old records
Lack of computer disc space made us use Place oriented way of saving data
Now these space restrictions are not relevant anymore
Fred lives on “Spring St.”
Place oriented storage
| Name | Street |
| fred | “Spring St.” |
Place oriented storage
fred moves to “Maple St.”
| Name | Street |
| fred |
Now we’ve lost information about Fred ever lived on “Spring St.”
Place oriented storage
| Name | Street |
| fred | “Maple St.” |
Facts oriented storage
| fred | :street | “Spring St.” | T15 |
fred moves to “Maple St.”
| fred | :street | “Maple St.” | T36 |
+----------------------------------------------------------------------------+ | +------------------------------------------------------------------------+ | | | Peer Application Process | | | +-------+--------------+-------------+------------------+----------------+ | | | ^ ^ ^ | | | | D| | | | | | a| | | | | | t| | | | V | a| | | | +-------+--------------+--------+ | +-------------+----------------+ | | | Transactor | S| | Memcached Cluster | | | +--------------+----------------+ e| +-------------+----------------+ | | | g| ^ | | | m| | | | | e| | | | | t| | | | V s| | | | +--------------+---------------------+------------------+----------------+ | | | Storage Service (JDBC SQL || Cassandra || DynamoDB) | | | +------------------------------------------------------------------------+ | +----------------------------------------------------------------------------+
Architecture Overview
1977 Datalog was created
Syntactically it is a subset of Prolog
Pets shelter database
Query: Owner names of dog “Tiny”
'[:find ?owner-name
:where [?dog :dog/name "Tiny"]
[?owner :owner/dogs ?dog]
[?owner :owner/name ?owner-name]]Example from Conversations With Datomic article by Carin Meier
- Query consistent database value at any moment in time
- Query historic data
- Make speculative What If queries
- Use user defined functions and rules in queries
- Use user defined functions on Transactor for data integrity
- Listen to all changes in the system and react
- Lookup indexes directly for speed
- Graph database
- Fulltext search
- InMemory storage for “UNIT” tests!!!
- Bitemporal quieries on transaction time and valid time
- Schema on read, schemaless on write
- EDN database
- Open Source (MIT Licenced)
- Distributed with all data on each node
- Data Persistence
- Cache Invalidation
- ETL from OLTP to OLAP
- Integration (Not so much you can do about that)
- SPA Stateful Client
- DB related Concurrency
- Scaling writes and reads separately
| Impedance Mismatch | -> Pull API |
| Cache Invalidation | -> Immutability |
| DB related Concurrency | -> Single Threaded Writes |
| ETL from OLTP to OLAP | -> as-of, since, history |
| Integration ( “Microservices” ) | -> Client, REST, Vase |
| SPA Client State Management | -> REST SSE, DataScript |
| Simulation Testing | -> Simulant |
| Scaling writes/reads separately | -> Datomic Architecture |
Datomic REST API works but considered legacy
That’s why it is the best choice to implement your own ideas and projects.
No incidental complexity, no restrictions, no crutches, no pain.
In fact you have Event Sourcing and CQRS with no cost.
Event Sourcing pattern is just a crutch for an outdated place oriented data model.
If data worth saving, all historic data worth saving as well.
You will do it anyway in OLAP database.
Any business application downloads hot dataset in memory
But Datomic doing it smarter, giving you a lot of benefits
And always gonna be closed source
But it was developed by Rich Hickey and Stuart Halloway
And it is totally free for one year of updates
If you are using proprietary DBs for OLTP, ETL and OLAP
You can afford using Datomic
Or consider using XTDB
Even thogh Clojure language was created to build Datomic
Datomic supports all JVM languages
Non JVM languages supported via REST API
Datalog is applicable for any structured data processing.
But Datomic uses Datalog to process Facts (Datoms).
If you don’t need history and only need to make requests for application state you can use Open Source Datalog implementation DataScript
For example, you can use DataScript to:
- Store and query Single Page Application’s State
- Statically generate site from Knowledge Base on DataScript
I want to inspire people to:
- Consider ideas Datomic based on
- Learn Datomic or XTDB and Datalog
- Use Datomic, XTDB and DataScript for pet projects and ambitious production
- Develop job market without data related incidental complexity
- Quit tolerate unnecessary incidental complexity
Videos from ClojureCourse.by by Nikita Prokopov
Interactive course Learn Datalog Today
Articles by Carin Meyer Conversations With Datomic
Great Datomic Blog and Documentation
https://nextjournal.com/xtdb-tutorial/start
Николай Рыжиков - за знакомство с Datomic и Clojure в 2013, знакомство с Reveal.js и название доклада
Никита Прокопов - за преподавание на ClojureCource.by в 2014 и Open Source DataScript и RUM
Сергей Ткаченко - за Moscow Clojure/CLJS Meetup и приглашение сделать доклад
Rich Hickey - за Clojure и Datomic
Stuart Halloway - За Datomic, Simulant и мою первую прочитанную книгу по Clojure Programing Clojure
GNU Emacs, Org Mode, Org-Reveal, Artist Mode, Prelude
Defn Episode 23 with Stuart Halloway about history of Cognitect and Datomic past and future
Cognicast Episode 059 With Michael Nygard about simulation testing
Cognicast Episode 101 With Luke Vanderhart about Arachne web framework wich uses Dataomic in memory database for components configuration
Cognicast Episode 118 with Paul deGrandis about Vase data-driven microservices library