Datomic Database Plugin

This is a plugin for Fulcro RAD that adds support for using Datomic databases as the back-end technology.

Note	The current version supports on-prem Datomic Pro (PostgreSQL, MySQL, or Microsoft SQL Server backends), Datomic Cloud, and Datomic-Local, as well as the nowadays deprecated Datomic Free.

Make sure you do not accidentally get Datomic Free and Datomic Pro BOTH on the CLASSPATH. If you’re using deps you should make sure you use something like:

{:deps {com.fulcrologic/fulcro-rad-datomic {:mvn/version "x.x.x"
                                            :exclusions  [com.datomic/datomic-free]}}}

Conventions

The following namespace aliases are used in the content of this document:

(ns x
  (:require
     [com.fulcrologic.rad.attributes :as attr]
     [com.fulcrologic.rad.database-adapters.datomic :as datomic]
; or [com.fulcrologic.rad.database-adapters.datomic-cloud :as datomic] for Datomic Cloud support
     [com.fulcrologic.rad.database-adapters.datomic-options :as do]))

Schemas

It is common for larger applications to desire some kind of sharding of their data, particularly in cases of high write load. The Datomic plugin has you model attributes on a Datomic schema which can then be applied to any number of runtime databases in your application. Of course, you must have some scheme for selecting the correct runtime database for mutations and read resolvers during operation.

Attribute Facets

Every attribute in the system that is stored in Datomic using this plugin must include a ::attr/schema <k> entry, where <k> is a keyword representing a schema name. All attributes that share the same schema name will be stored together in a database that has that schema (see Selecting a Database During Operation).

The following common attribute keys are supported automatically:

::attr/unique?

Causes the attribute to be a unique value, unless a Datomic-specific override for uniqueness is supplied or the attribute is marked as an ::attr/identity?.

::attr/identity?

Causes the attribute to be a unique identity. If supplied, then ::attr/unique? is assumed.

::attr/cardinality

:one or :many (or you can specify it with normal datomic keys).

::attr/schema keyword

(required) A name that groups together attributes that go together in a schema on a database. A schema can be used on any number of databased (e.g. for sharding).

::attr/identities #{k k2 …​}

Required on non-identity attributes. A set of attribute keys that are :attr/identity? true. This set indicates that the attribute can be placed on an entity that is identified by one of those identity attributes. This allows the Datomic plugin to figure out which properties can be co-located on an entity for storage in form saves and queries in resolvers. It is a set in order to support the fact that Datomic allows an attribute to appear on any entity, but your domain model will put it on a more limited subset of different entities. Failing to list this entry will result in failure to generate resolvers or form save logic for the attribute.

The plugin-specific attribute parameters are:

do/native-id?

If true it will map the given ID attribute (which must be type long) to :db/id.

do/attribute-schema

A map of datomic schema attributes to be included in transacted schema. example: {:db/isComponent true}

Any of the normal Datomic schema definition attributes can be included as well (e.g. :db/cardinality), and will take precedence over anything above. Be careful when changing these, as the result can cause incompatible change errors.

Automatic Generation

During development you may choose to create a transaction that can be used to create/update the schema in one or more Datomic databases. This feature is will work as long as your team follows a strict policy of only ever making compatible changes to schema (e.g. mostly additions).

This feature is great for early development, but it may become necessary over time to adopt a migration system or other more manual schema management policy. This feature is therefore easy to opt in/out of at any time.

You can pull the current generated schema from the attribute database using (datomic/automatic-schema schema-key). NOTE: You must require all namespaces in your model that define attributes to ensure that they are all included in the generated schema.

Enumerations

The idiomatic way to represent enumerations in Datomic is with a ref and a set of entities known by well-known idents. The following is supported during automatic schema generation:

(new-attribute :address/state :enum
  {:attr/enumerated-values #{:AL :address.state/CA {:db/ident :address.state/OR :db/doc "Oregon"}}})

All three of the above forms are allowed. An unqualified keyword will be auto-qualified (AL and CA above could be represented either way), and a map will be treated like a full-blown entity definition (will be passed through untouched to the schema).

Validation

If you are not using automatic schema generation then it is recommended that you at least enable schema validation. This feature can be used to check the schema of an existing database against the current code of your attributes to detect inconsistencies between the two at startup. This ensures you will not run into runtime errors due to code that does not match the schema of the target database.

TODO: Write the validator, and document it.

Databases

It is up to you to configure, create, migrate, and manage your database infrastructure; however, this plugin comes with various utilities that can help you set up and manage the runtime environment. You must follow certain conventions for things to work at all, and may choose to opt into various features that make it easy to get started and evolve your system.

Runtime Operation

Mocked Connections during Development

The Datomock library is a particularly useful tool during experimental phases of development where you have yet to stabilize a particular portion of schema (attribute declarations). It allows you to "fork" a real database connection such that any changes (to schema or otherwise) are thrown away on application restarts.

This allows you to play with new schema without worrying about incompatible schema changes.

It is also quite useful for testing, since it can be used to pre-create (and cache) an in memory database that can be used to exercise Datomic code against your schema without the complete overhead of starting an external database with new schema.

Selecting a Database During Operation

When you set up your Pathom parser you can provide plugins that modify the environment that will be passed by Pathom to all resolvers and mutations on the server. The generated resolvers and mutations for the Datomic plugin need to be able to decide which database should be used for a particular schema in the context of the request. Atomic consistency on reads requires that such a database be provided as a value, whereas mutations will need a connection.

The env must therefore be augmented to contain the following well-known things:

::datomic/connections - A map, keyed by schema, of the database connection that should be used in the context of the current request. ::datomic/databases - A map, keyed by schema, of the most recent database value that should be used in the context of the current request (for consistent reads across multiple resolvers).

TODO: Supply helper functions that can help with this

Indexed Access (Version 1.3.1+)

Prior versions had experimental versions of approaches to fast indexed access for RAD reports. Version 1.3.1 and above have a stabilized version of this that works well with the RAD report server-paginate-report UI state machine. This support includes:

• A function that can grab a page of sorted/filtered data from a tuple tuple-index-scan using a range and filters against an datomic/index-range.

• A helper function for converting RAD report search parameters into parameters acceptable for use with tuple-index-scan.

• A function to generate a resolver that can use a tuple plus d/index-range to quick find a page of data, and which conforms to the new server-paginated-report machine in RAD.

Testing

Custom mutations and resolvers are easiest to write if you have a simple way of testing them against a database that looks like your real one. This plugin supports some helpful testing tools that leverage Datomock to give you a fast an consistent starting point for your tests.

Seeding Development Data

We recommend using UUID domain IDs for all entities (e.g. :account/id). This not only enables much of the resolver logic, it also allows you to easily and consistently seed development data for things like live coding and tests.

The com.fulcrologic.rad.ids/new-uuid function can be used to generate a new random UUID in CLJC, but it can also be used to generate a constant (well-known) UUID for testing.

A Sample Test

If you are using on-prem, the core function to use is datomic/empty-db-connection, which can work with automatically-generated schema or a manual schema. It returns a Datomic connection which has the supplied schema (and is memoized for fast startup on sequences of tests).

A typical test might look like the following:

(deftest sample-test
  ;; the empty-db-connection can accept a schema txn if needed.
  (let [conn        (datomic/empty-db-connection :production)
        sample-data [{::acct/id   (new-uuid 1)
                      ::acct/name "Joe"}]]
    @(d/transact conn sample-data)

    (let [db (d/db conn)
          a  (d/pull db [::acct/name] [::acct/id (new-uuid 1)])]
      (is (= "Joe" (::acct/name a))))))

Note	The connection is memoized based on the schema key (not any supplied migration data). You can use (datomic/reset-test-schema k) to forget the current memoized version.

For Datomic Cloud, the current recommendation is to use the com.datomic/local with the :mem storage. See the fulcrologic/fulcro-rad-demo for an example.

Resolver Generation

Contributing to This Library

We use git (with git flow) for source control. Please branch and make PRs against the develop branch.

There is an example application fulcrologic/fulcro-rad-demo that can be used when developing features.

You will need Datomic Pro with a PostgreSQL or MySQL backend or Datomic Free to run the example. Follow the instructions for setting that up, and then edit the defaults.edn file in src/example/config and update the database parameters to match your system.

 :com.fulcrologic.rad.database-adapters.datomic/databases
    {:main {:datomic/schema           :production
            :datomic/driver           :postgresql ;; OR :mysql :sqlserver :free :mem
            :datomic/database         "example"
            :datomic/prevent-changes? true
            :postgresql/host          "localhost"
            :postgresql/port          5432
            :postgresql/user          "datomic"
            :postgresql/password      "datomic"
            :postgresql/database      "datomic"
            :free/host                "localhost"
            :free/port                4334}}

LICENSE

The MIT License (MIT) Copyright (c), Fulcrologic, LLC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.