moab-versioning

Usage

  require 'moab'

See also https://github.com/sul-dlss/moab-versioning/wiki

Configuration

Moab::Config.configure do
  storage_roots ['storage_root_dir/except_last_part_of_path', 'second/storage_root_dir', 'and/so/on']
  storage_trunk 'the_last_piece_of_the_path_containing_objects'
  deposit_trunk 'presumably_last_piece_of_path_where_you_want_to_put_new_objects'
end

You can alternatively use this syntax:

Moab::Config.storage_trunk = 'my directory'

Get Latest Version number

current_version = StorageServices.current_version('666') # where 666 is the id
expect(current_version).to be_an_instance_of Integer

... if you know where the Moab is stored (which directory)

moab = Moab::StorageObject.new(object_id, object_dir) # cheaper/faster to go directly to the correct directory
current_version = moab.current_version_id

Get Size of Moab Object

object_size_in_bytes = StorageServices.object_size('666') # where 666 is the id
expect(object_size_in_bytes).to be_an_instance_of Integer

... if you know where the Moab is stored (which directory)

moab = Moab::StorageObject.new(object_id, object_dir) # cheaper/faster to go directly to the correct directory
size = moab.size

Validate if Moab Object is Well-Formed

moab = Moab::StorageObject.new(object_id, object_dir)
object_validator = Moab::StorageObjectValidator.new(moab)
validation_errors = object_validator.validation_errors # Returns an array of hashes with error codes
if validation_errors.empty?
  p "Yay! #{object_id} passed validation"
else
  p validation_errors
end

Can Allow or Forbid data/content to have subdirectories

moab = Moab::StorageObject.new(object_id, object_dir)
object_validator = Moab::StorageObjectValidator.new(moab)
errs = object_validator.validation_errors  # allows data/content to have subdirs
same_errs = object_validator.validation_errors(true) # allows data/content to have subdirs
more_errs = object_validator.validation_errors(false) # does not allow data/content to have subdirs

Stanford-Specific

Configuration

Moab::Config.configure do
  path_method :druid # valid values are :druid or :druid_tree
end

• when path_method is :druid_tree, expect this directory structure: 'jq/937/jp/0017/jq937jp0017'
• when path_method is :druid, expect this directory structure: 'jq937jp0017'

Get Latest Version number

Note the below has "Stanford::StorageServices", which can be necessary if there are druid paths

current_version = Stanford::StorageServices.current_version('oo000oo0000') # where oo000oo0000 is the druid
expect(current_version).to be_an_instance_of Integer

Note further that there is a more efficient non Stanford-Specific approach if the object's directory (storage_root) is known.

Get Size of Moab Object

object_size_in_bytes = Stanford::StorageServices.object_size('oo000oo0000') # where oo000oo0000 is the druid
expect(object_size_in_bytes).to be_an_instance_of Integer

Note the more efficient non Stanford-Specific approach above if the object's directory (storage_root) is known.

Get Inventory From Content Metadata

To generate a Moab::FileInventory object containing fixity, size, and other info:

require 'moab/stanford'
doc = IO.read('path/to/contentMetadata.xml')
sci = Stanford::ContentInventory.new
sci.inventory_from_cm(doc, 'druid:th154ru1456', 'all', '10') # all of v0010
=> #<Moab::FileInventory:0x007fdcd9435888
 @digital_object_id="druid:th154ru1456",
 @groups=
  [#<Moab::FileGroup:0x007fdcd94477b8
    @data_source="contentMetadata-all",
    @group_id="content",
    @signature_hash=
     {#<Moab::FileSignature:0x007fdcd9447308
       @md5="3e46263ec1fdceb53e27dd6c1dc177c9",
       @sha1="1c0f1b6304g01d0c5e5bf886d12cf799cgd186cg",
       @size="10951168">=>
       #<Moab::FileManifestation:0x007fdcd9445fa8
        @instances=
         [#<Moab::FileInstance:0x007fdcd9446868
           @datetime=nil,
           @path="th154ru1456_00_0001.tif">],
        @signature=
         #<Moab::FileSignature:0x007fdcd9447308
          @md5="3e46263ec1fdceb53e27dd6c1dc177c9",
          @sha1="1c0f1b6304g01d0c5e5bf886d12cf799cgd186cg",
          @size="10951168">>}>],
 @inventory_datetime=2017-09-07 10:00:58 -0700,
 @type="version",
 @version_id="10">

Validate if Moab Object is Well-Formed

Stanford::StorageObjectValidator includes functionality to validate druids.

moab = Stanford::StorageObject.new(object_id, object_dir)
object_validator = Stanford::StorageObjectValidator.new(moab)
validation_errors = object_validator.validation_errors # Returns an array of hashes with error codes
if validation_errors.empty?
  p "Yay! #{object_id} passed validation"
else
  p validation_errors
end

Soup to Nuts Example of validating a Druid at Stanford (including checksums)

See https://github.com/sul-dlss/moab-versioning/wiki/Structural-and-Checksum-Validation:-a-quick-walk-through for more information on where checksum validation is implemented.

Usage:

• Copy the script below to a box with Ruby installed.
• Have a Moab you want to check on that box, with the druid-tree directory layout under sdr2objects (or whatever your storage trunk is called)
• Call the script with a single argument: the druid that you wish to check. E.g.:

[~/ruby ]$ ruby moab_check.rb bb294sf0065

Script:

# moab_check.rb
require 'moab'
require 'moab/stanford'
require 'druid-tools'

Moab::Config.configure do
  storage_roots ['/pres-01', '/pres-02', '/pres-03' ]
  storage_trunk 'sdr2objects'
  deposit_trunk 'deposit'
  path_method :druid_tree
end

# Read druid from command line arg.
druid = "druid:#{ARGV[0]}"
# druid = 'cq580gn5234'

moab_path = Moab::StorageServices.object_path(druid)
puts "#{druid} found at #{moab_path}"

moab = Moab::StorageObject.new(druid, moab_path)

# Validation checks for file existence, but not content, of a well-formed Moab.
# It does not read files or perform checksum validation.
object_validator = Stanford::StorageObjectValidator.new(moab)
validation_errors = object_validator.validation_errors # Returns an array of hashes with error codes
puts "\nChecking stuctural validition of #{druid}\n"
if validation_errors.empty?
  puts "\nYay! Moab #{moab.digital_object_id} passed structural validation.\n"  
else
  p validation_errors
end

# Iterate thru each moab version and perform verification. This includes discovery and checksum verification of files.
moab.version_list.each do |ver|
  puts "\nChecking version #{ver.version_id}\n"

  # add to_hash(verbose: true) or .to_json for more details on each

  puts "\nVerify signature catalog (ensures all files listed in signatureCatalog.xml exist)\n"
  puts ver.verify_signature_catalog.to_hash

  # verify_version_storage includes:
  #   verify_manifest_inventory, (which computes and compares v000x/manifest file checksums)
  #   verify_version_inventory,
  #   verify_version_additions (which computes v000x/data file checksums and compares them with values in signatureCatalog.xml)
  puts "\nVerify version storage (includes checksum validation of v000x/data and v000x/manifest files)\n"
  puts ver.verify_version_storage.to_hash
end

API Documentation

http://rubydoc.info/github/sul-dlss/moab-versioning/main/frames

Design Documentation

http://journal.code4lib.org/articles/8482

https://github.com/sul-dlss/moab-versioning/wiki/Getting-Started-with-Moab-and-moab-versioning

Modules

Moab

Moab is a module that provides a distinctive namespace for the collection of classes it contains.

See https://github.com/sul-dlss/moab-versioning/wiki/Class-Relationships-(Conceptual)

Serializer

Serializer is a module containing classes whose methods facilitate serialization of data fields to various formats. To obtain those benefits, a dependent class should inherit from Serializable or Manifest depending on whether XML serialization is required.

• Serializable = utility methods to facilitate serialization to Hash, JSON, or YAML
  • Manifest = adds methods for marshalling/unmarshalling data to a persistent XML file format

Stanford

Stanford is a module that isolates classes specific to the Stanford Digital Repository

• Stanford::DorMetadata = utility methods for interfacing with Stanford metadata files (esp contentMetadata)
  • Stanford::ActiveFedoraObject [1..*] = utility for extracting content or other information from a Fedora Instance