ENDIT daemons 1.3.0

ENDIT daemons 1.3.0

This release is a bugfix and improvement release:

• tsmdeleter now queues deletes to periodically process them, this batches deletes that trickles in over a longer time period which improves the efficiency of tape space reclamation.
  • NOTE that this means that the tape pools now have state for deleted files as well, see the README for details on pool decommission etc.
• tsmtapehints now respects the ENDIT_RETRIEVER_HINTFILE environment variable to override the hintfile location.
• tsmarchiver now spawns multiple single-drive sessions instead of a single session with varying resource usage. This works around weird corner cases where activity halts pending resources such as a tape drive becoming available. This also allows a quicker reaction to load change, and a better threshold based strategy better adapts to incoming data rate.
  • A side effect of this is obsoleting the configuration options archiver_threshold[1-9]_dsmcopts. They can simply be removed from your configuration.
• The tsmretriever is now aware the extended metadata produced by newer ENDIT dCache plugin versions, and doesn't try to spawn restores for archive session data.

In addition to this are assorted smaller fixes. The complete list of changes since the last release can be viewed on the GitHub compare page

For additional information and installation instructions, see the README at https://github.com/neicnordic/endit/blob/master/README.md

ENDIT daemons 1.2.1

ENDIT daemons 1.2.1

This release is a minor bugfix and improvement release:

• Avoid syntax that causes perl 5.24 and older to emit warnings.
• Keep actual error messages when logging verbose dsmc error output.
• Handle USR1 signal to skip delay timers.

In addition to this are assorted smaller fixes. The complete list of changes since the last release can be viewed on the GitHub compare page

For additional information and installation instructions, see the README at https://github.com/neicnordic/endit/blob/master/README.md

ENDIT daemons 1.2.0

ENDIT daemons 1.2.0

This release is a major improvement release:

• ENDIT daemons now stores files to tape ordered by age. In the use cases where files are retrieved with the same grouping as they were written this improves retrieve performance. Work on this issue revealed that the IBM Spectrum Protect (TSM) client actually sorted the file list and stored files in alphabetical order and by that undoing any attempts by ENDIT daemons or others at ordering or grouping files optimally. This has been addressed by IBM in IBM APAR IT33143 and you are recommended to install client version 8.1.11.0 or later.
• Other improvements include making verbose logging be more talkative while waiting for data when below thresholds and updates in example configuration and README.

In addition to this are assorted smaller fixes. The complete list of changes since the last release can be viewed on the GitHub compare page

For additional information and installation instructions, see the README at https://github.com/neicnordic/endit/blob/master/README.md

ENDIT daemons 1.1.1

ENDIT daemons 1.1.1

This release is a minor bugfix release:

• Fix issue in the TSM server side tape hint file generator tsm_getvolumecontent.pl that caused unavailable files to not be present in the tape hint file, which in turn had the side effect of frequent retries and logs filling up should those files be requested.

In addition to this are assorted smaller fixes. The complete list of changes since the last release can be viewed on the GitHub compare page

For additional information and installation instructions, see the README at https://github.com/neicnordic/endit/blob/master/README.md

ENDIT daemons 1.1.0

ENDIT daemons 1.1.0

This release has seen extensive validation in the form of production use on the NDGF Tier1
and is regarded as superior to all previous ENDIT versions.

A number of significant improvements have been made since release 1.0.0:

• The description added to the archived files now includes the time of the archive operation. This helps debugging on the IBM Spectrum Protect (TSM) level.
• It is now possible to create tape hint files on the pool node using tsmtapehints.pl, ie. without TSM administrative user access.
  • A bonus feature implemented in tsmtapehints.pl is the detection of duplicated copies of archived files, this is typically caused by bugs in older ENDIT daemon versions. See #12 for information on how to handle duplicates.
• Tape hint files are now written in JSON format. Reading of the old text format is still supported, but will be removed in a future release.
  • As a side effect of this we strongly recommend the JSON::XS perl module to be installed for good performance.
• Use of the IPC::Run3 perl module has been eliminated.
• Signal handling has been improved and all child processes should now die promptly when the main process is killed.
• Daemons now does chdir / on startup to avoid command execution issues if the current directory is removed after startup.

In addition to this are assorted smaller fixes. The complete list of changes since the last release can be viewed on the GitHub compare page

For additional information and installation instructions, see the README at https://github.com/neicnordic/endit/blob/master/README.md

ENDIT daemons 1.0.0

First public release

This is the first public release of the ENDIT daemons. While it works fine in production on one site, we expect minor issues with items such as documentation, suggested configuration defaults etc.