Weaver

Weaver: Clone n-ary relationship sets across distinct databases.

NPM package: @skelogh/weaver

Context

• Entity-relationship model
• Extended reference pattern

Debugging DB objects with n-ary relationships is often done manually, this tool is meant to automate copying these objects into desired databases.

For example, a database has the following collections/documents:

  // db.users.findOne({_id: ObjectId('abcdef78901234abcdef1234')})
  {
    _id: ObjectId('abcdef78901234abcdef1234'),
    name: 'John',
    orders: [ { orderId: '4321fedcbafedcba67890123' } ]
  }

  // db.orders.findOne({_id: ObjectId('4321fedcbafedcba67890123')})
  {
    _id: ObjectId('4321fedcbafedcba67890123'),
    cartId: 'fedcba67890123fedcba4321'
  }

  // db.carts.findOne({_id: ObjectId('fedcba67890123fedcba4321')})
  {
    _id: ObjectId('fedcba67890123fedcba4321'),
    userId: 'abcdef78901234abcdef1234'
  }

Note how user relates to order, and order relates to cart.

If you wanted to migrate the orders and carts associated to the user to your local environment, you'd need to:

1. Go to the [users|orders|carts] collection and find the document.
2. Check if any of the fields is a reference to another collection.
3. Copy the reference value.
4. Repeat from step 1.

This tool finds the relationships, and migrates them to a destination db:

Query	source db	target db
db.users.findOne(ObjectId('abcdef78901234abcdef1234'))	1	1
db.orders.findOne(ObjectId('4321fedcbafedcba67890123'))	1	1
db.carts.findOne(ObjectId('fedcba67890123fedcba4321'))	1	1

Usage

Requirements

• nvm 0.33.
• node v16.18.0.
• npm 6.4.1.

Getting started

Weaver runs on NodeJS >= 10

Install via npm

npm i -g @skelogh/weaver

How to test the CLI in dev mode

Install the CLI tool

• After cloning the repo and installing dependencies, run npm run build.
• Run npm pack on the project root.
• A file weaver-<VERSION>.tgz will be created.
• Run npm install -g weaver-<VERSION>.tgz, it will install the package globally.

Using the CLI tool

• After installing, just issue the weaver command, the CLI help will display the CLI help:

Usage: weaver [OPTIONS] COMMAND [ARG...]
       weaver [ --help | -v | --version ]

Commands:
  weaver add        Creation of client, query or ignore
  weaver remove     Removal of clients, queries or ignores
  weaver run        Runs the app with the loaded configuration

Options:
  --version, -v    Print version information and quit
  --config, -c     Read or set path of config file, default: undefined
  --queries, --qq  Document ids to get relationships from, e.g.: 2a3b4c5d6e7f8g9h2a3b4c5d e7f8g9h2a3b4c5d2a3b4c5d6
  --verbose, -V    Enable highest level of logging, same as DEBUG=*
  --help           Show help

1. Add clients

In the CLI, run weaver add client, the following help will display:

Usage: weaver add client -fntou --options.<option_name>

  -f [mongodb]          <String> The client db family, mongodb is only supported for now.
  -n <name>             <String> The client db name.
  -t [source|target]    <String> source if the data will be pulled from it, target otherwise.
  -o [<source.name>]    <String> The target's origin db where the data will be copied from.
  -u <url>              <String> The client db URL.
  --options.<opt_name>  <Any> Client db-specific options, for now MongoClient options, use dot notation to set each option
                        Example: --options.readPreference secondaryPreferred

To pull data from a db into another, you'll need to add both the source and target clients. The source is where the document we want is stored, and target is where you want to copy to.

Example:

Assume the following remote server settings for source:

• Family: mongodb
• Type: source
• IP: 172.17.0.4
• Port: 27017
• Database name: production

weaver add client -f mongodb -n production -t source -u mongodb://172.17.0.4:27017

Assume the following local server settings for target:

• Family: mongodb
• Type: target
• IP: 127.0.0.1
• Port: 27017
• Database name: development
• Origin: production (This is used to map where the data will be pulled from, useful when dealing with multiple source and target databases.)

weaver add client -f mongodb -n development -t target -o production -u mongodb://127.0.0.1:27017

2. Find the document to clone

From the source database, find the document you want to copy to the target database and copy the stringified _id:

// Assume the following mongodb document in the `production` database, `pets` collection:
{
    _id: ObjectId("507f1f77bcf86cd799439011"),
    name: "fluffy"
}

3. Clone the document(s)

Once steps 1 & 2 are done, just run the following command:

weaver run --queries 507f1f77bcf86cd799439011

4. See the results

If all is setup correctly, you should now have a copy of the document in your target database, it will create the collection even if it didn't exist before:

> mongodb
mongodb> use development
mongodb> db.pets.find(ObjectId("507f1f77bcf86cd799439011"))

{
    _id: ObjectId("507f1f77bcf86cd799439011"),
    name: "fluffy"
}

>

Roadmap

• weaver CLI (follow the project)
  • npm i -g @skelogh/weaver 🎉
• Add plugins for different data sources.
  • MongoDB
  • [WIP] ElasticSearch (follow the project)
  • DinamoDB
  • Cassandra
  • and so on...
• Client UI
• API/SDK (?)
• Connect via SSH.
• Doc pages

See what's been worked on.

Questions? create an issue!.