CMDB

CMDB is a Ruby interface for consuming data from one or more configuration management databases (CMDBs) and making that information available to Web applications.

It is intended to support multiple CM technologies, including:

• consul
• JSON/YAML files on a local disk
• (someday) etcd
• (someday) ZooKeeper

Maintained by

• RightScale Inc.

Why should I use this gem?

With CMDB, you can:

1. Decouple your modern (12-factor) application from the CM tools used to deploy it
2. Deploy legacy applications by rewriting config files at app-load time
3. Interact with your CMDB using a beautiful shell

CMDB has three primary interfaces:

1. The cmdb shell command navigates your k/v store using filesystem-like metaphors (ls, cd, and so forth) with color output and tab completion
2. The cmdb shim command populates the environment with values and/or rewrites hardcoded config files, then spawns your application.
3. The CMDB::Interface object provides a programmatic API for querying CMDBs. Its #to_env method transforms the whole configuration into an environment-friendly hash if you prefer to seed the environment yourself, without using the shim.

Getting Started

Determine sources

Sources are specified with the --source option when you run the CLI. This option applies to all subcommands (shim, shell, etc) and must appear before the subcommand name.

You can add as many sources as you'd like. All sources are specified as a URI, where the scheme tells CMDB which driver to use and how to interpret the rest of the URI.

Examples:

• file:///var/lib/cmdb/myapp.yml creates a file source with the prefix myapp; the value foo: bar in the file would have the key myapp.foo
• consul://localhost creates a source with no key prefix that talks to a local consul agent on the standard port (8500); a value foo/bar in Consul would have the key `foo.bar
• consul://kv:18500/myapp creates a source with the prefix myapp. that talks to a remote consul agent on a nonstandard port (18500); this source only "sees" Consul values under the path /myapp/ and their key names always begin with myapp.
• consul://localhost/mycorp/staging/myapp creates a source with the prefix myapp.; this source only "sees" Consul values under the path staging/myapp and their key names always begin with myapp.
• consul://localhost/mycorp/staging creates a source with the prefix staging. that has all keys in the staging environment. (It is probably a bad idea to use this source with the myapp source in the example above!)

If no sources are specified on the command line, CMDB will run an auto-detect algorithm to check for network agents listening at localhost.

To learn more about sources and prefixes, see "Data model," below.

Invoke the CMDB Shell

To enter an interactive sh-like shell, just type cmdb shell.

Invoke the CMDB Shim

For non-Ruby applications, or for situations where CMDB values are required outside of the context of interpreted code, use cmdb shim to run your application. The shim can do several things for you:

1. Make CMDB values available to your app (in ENV, or by rewriting files)
2. Change the user before invoking your app (e.g. drop privileges to www-data)
3. Watch for filesystem changes and reload your app on demand

Populate the environment for a dotenv-compatible application

If you have an app that uses 12-factor (dotenv) style configuration, the shim can populate the environment with CMDB values:

bundle exec cmdb shim

# Now your app can refer to ENV['DB_HOSTNAME'] or ENV['WIDGETS_FLAVORS]
# Note missing "my_app" prefix that would be present if you asked for these using their CMDB key names

Note that when we export CMDB keys into the environment, we remove the prefix of each key; in the example above, the values could have come from common.db.hostname and myapp.widgets.flavors but their names have been simplified. If any two sources have keys whose simplified names are identical, CMDB prints a detailed error message and fails rather than putting ambiguous data into the environment.

Note that the data type of CMDB inputs is preserved: lists remain lists, numbers remain numbers, and so forth. This works irrespective of the format of your configuration files, and also holds true for CMDB values that are serialized to the environment (as a JSON document, in the case of lists).

Rewrite static configuration files with dynamic CMDB values

If the --rewrite option is provided, the shim recursively scans the provided subdirectory for data files that contain replacement tokens; when a token is found, it substitutes the corresponding CMDB key's value.

Replacement tokens look like this: <<name.of.my.key>> and can appear anywhere in a file as a YAML or JSON value (but never a key). Unlike environment variables, replacement tokens always use the fully-qualified key name, including prefix.

Replacement tokens should appear inside string literals in your configuration files so they don't invalidate syntax or render the files unparsable by other tools.

The shim performs replacement in-memory and saves all of the edits at once, making the rewrite operation nearly atomic. If any keys are missing, then no files are changed on disk and the shim exits with a helpful error message.

Given my_app.yml and an application with two configuration files:

# config/database.yml
production:
  host: <<my_app.db.hostname>
  database: my_app_production

# config/widgets.json
{'widgetFlavors': '<<my_app.widgets.flavors>>'}

I can run the following command in my application's root directory:

bundle exec cmdb shim --dir=config rackup

This will rewrite the files under config, replacing my configuration files as follows:

# config/database.yml
production:
  host: db1.local
  database: my_app_production

# config/widgets.json
{'widgetFlavors':['vanilla', 'chocolate']}

Drop Privileges

If your app doesn't know how to safely switch to a non-privileged user, the shim can do this for you. Just add the --user flag when you invoke it:

bundle exec cmdb shim --user=www-data whoami

Data Sources

Network Servers

To read CMDB data from a consul server, add a CLI parameter such as --source=consul://some-host/key/subkey. This will create a source whose prefix is subkey that encompasses the subtree of the k/v store that lies underneath /key/subkey.

Flat Files

To read CMDB data from a flat file on disk, add a CLI parameter such as --source=file:///var/lib/cmdb/mykeys.yml. This will parse the YAML file located in /var/lib/cmdb and present it as a source whose prefix is mykeys.

JSON and YAML files are both supported. The structured data within each file can contain arbitrarily-deep subtrees which are interpreted as subkeys, sub-subkeys and so forth.

Data Model

CMDB models all data sources as trees whose nodes are named, and whose leaf nodes can contain a piece of data: strings, numbers, booleans, or arrays.

Maps are disallowed on order to prevent ambiguity; a map always represents a subtree of the k/v store, never a value. Mixed-type arrays are disallowed because they can cause problems with strongly-typed languages. Nil is disallowed as a value because writing nil means "delete the key."

Paths within a tree -- and therefore CMDB keys -- are identified using a dot notation similar to Java properties; for instance, production.http.listen_port might be an Integer-valued key that tells your microservice which HTTP port to listen on in the production environment; production.database.host might be the database host, and so forth. The names of keys are determined by the tree structure of your k/v store, and when you set a key through CMDB, its position in the tree is derived from its key name.

Source Prefixes

CMDB sources have a prefix, meaning that all keys contained in that source begin with the same prefix. No two sources may share a prefix, ensuring that sources don't "hide" each others' data. The prefix of a source is usually automatically determined by the final component of its URL, e.g. the filename in the case of file:// sources and the final path component in the case of consul:// or other network sources.

Inheritance

The uniqueness constraint on prefixes means that all sources' keys are disjoint; there is no such thing as "inheritance" in the CMDB data model.

When keys are exported to the environment, the prefix is stripped from the key name; however, CMDB still prevents overlap in this case.

Inheritance may be supported in future as an optional behavior, but is omitted for the time being because in practice, it causes more problems than it solves.

Ambiguous Key Names

Consider a file that defines the following variables:

# confusing.yml
this:
  is:
    ambiguous
  was:
    very: ambiguous
    extremely: confusing

At first glance, ths file defines two CMDB keys:

• confusing.this.is (a string)
• confusing.this.was (a map)

However, an equally valid interpretation would be:

• confusing.this.is
• confusing.this.was.very
• confusing.this.was.extremely

Because CMDB keys cannot contain maps, the first interpretation is wrong. The second interpretation is valid according to the data model, but results in a situation where the type of the keys could change if the structure of the YML file changes.

For this reason, any YAML file that defines an "ambiguous" key name will cause an error at initialization time. To avoid ambiguous key names, think of your YAML file as a tree and remember that leaf nodes must define data and internal nodes must define structure.