Merge pull request #187 from pepkit/dev

release 0.12.1

docs/changelog.md

@@ -2,7 +2,23 @@
 
 This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format. 
 
-## [0.12.0] -- 2019-05-02
+## [0.12.1] -- 2019-05-20
+
+### Added
+- `Sample` derived from `peppy.Sample` that specifies functionality less tied to a `Sample`'s essence, more related to a submission context, and hence more relevant to `looper` than to `peppy`.
+
+### Changed
+- Status table creation is possible outside of `looper`.
+- In the summary index page the plottable columns list is now scrollable
+- Status page relies on the `profile.tsv` file rather than `*.log`; [Issue 159](https://github.com/pepkit/looper/issues/159)
+
+### Fixed
+- In HTML reporting module, do not ignore objects which are neither HTMLs nor images in the summary, e.g. CSVs
+- Restore parsing and application of pipeline-level computing resource specification from a pipeline interface file; [Issue 184](https://github.com/pepkit/looper/issues/184)
+- Allow `ignore_flags` to properly modulate submission messaging; [Issue 179](https://github.com/pepkit/looper/issues/179)
+- Do not display time-like summary columns as the plottable ones; [Issue 182](https://github.com/pepkit/looper/issues/182)
+
+## [0.12.0] -- 2019-05-03
 
 ### Added
 - First implementation of pipeline interface 'outputs', so pipeline authors can specify items of interest produced by the pipeline.
@@ -14,14 +30,15 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 
 ### Fixed
 - Prevent CLI option duplication in pipeline commands generated
+- Make functional CLI spec of particular attribute on which to base selection of a subset of a project's samples ([`peppy` 298](https://github.com/pepkit/peppy/issues/298))
 
-## [0.11.1] - 2019-04-17
+## [0.11.1] -- 2019-04-17
 
 ### Changed
 - Improved documentation
 - Improved interaction with `peppy` and `divvy` dependencies
 
-## [0.11] - 2019-04-17
+## [0.11] -- 2019-04-17
 
 ### Added
 - Implemented `looper rerun` command.
@@ -37,7 +54,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - Removed Python 3.4 support.
 - UI: change parameter names `in/exclude-samples` to `selector-in/exclude`.
 
-## [0.10.0] - 2018-12-20
+## [0.10.0] -- 2018-12-20
 
 ### Changed
 - `PipelineInterface` now derives from `peppy.AttributeDict`.
@@ -50,7 +67,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - Existence of an object lacking an anchor image is no longer problematic for `summarize`.
 - Basic package test in Python 3 now succeeds: `python3 setup.py test`.
 
-## [v0.9.2] - 2018-11-12
+## [v0.9.2] -- 2018-11-12
 
 ### Changed
 - Fixed bugs with `looper summarize` when no summarizers were present
@@ -59,44 +76,44 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - Added new display features (graphs, table display) for HTML summary output.
 
 
-## [0.9.1] - 2018-06-30
+## [0.9.1] -- 2018-06-30
 
 ### Changed
 - Fixed several bugs with `looper summarize` that caused failure on edge cases.
 
-## [0.9.0] - 2018-06-25
+## [0.9.0] -- 2018-06-25
 
 ### Added
 - Support for custom summarizers
 - Add `allow-duplicate-names` command-line options
 - Allow any variables in environment config files or other `compute` sections to be used in submission templates. This allows looper to be used with containers.
 - Add nice universal project-level HTML reporting
 
-## [0.8.1] - 2018-04-02
+## [0.8.1] -- 2018-04-02
 
 ### Changed
 - Minor documentation and packaging updates for first Pypi release.
 - Fix a bug that incorrectly mapped protocols due to case sensitive issues
 - Fix a bug with `report_figure` that made it output pandas code
 
 
-## [0.8.0] - 2018-01-19
+## [0.8.0] -- 2018-01-19
 
 ### Changed
 - Use independent `peppy` package, replacing `models` module for core data types.
 - Integrate `ProtocolInterface` functionality into `PipelineInterface`.
 
-## [0.7.2] - 2017-11-16
+## [0.7.2] -- 2017-11-16
 ### Changed
 - Correctly count successful command submissions when not using `--dry-run`.
 
-## [0.7.1] - 2017-11-15
+## [0.7.1] -- 2017-11-15
 
 ### Changed
 - No longer falsely display that there's a submission failure.
 - Allow non-string values to be unquoted in the `pipeline_args` section.
 
-## [0.7] - 2017-11-15
+## [0.7] -- 2017-11-15
 ### Added
 - Add `--lump` and `--lumpn` options
 - Catch submission errors from cluster resource managers
@@ -112,7 +129,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - Sample child classes are now defined explicitly in the pipeline interface. Previously, they were guessed based on presence of a class extending Sample in a pipeline script.
 - Changed 'library' key sample attribute to 'protocol'
 
-## [0.6] - 2017-07-21
+## [0.6] -- 2017-07-21
 ### Added
   - Add support for implied_column section of the project config file
   - Add support for Python 3
@@ -130,7 +147,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - Require `setuptools` for installation, and `pandas 0.20.2`. If `numexpr` is installed, version `2.6.2` is required.
 - Allows tilde in `pipeline_interfaces`
 
-## [0.5] - 2017-03-01
+## [0.5] -- 2017-03-01
 ### Added
 - Add new looper version tracking, with `--version` and `-V` options and printing version at runtime
 - Add support for asterisks in file paths
@@ -145,7 +162,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - More robust installation and more explicit requirement of Python 2.7
 
 
-## [0.4] - 2017-01-12
+## [0.4] -- 2017-01-12
 ###  Added
 - New command-line interface (CLI) based on sub-commands
 - New subcommand (`looper summarize`) replacing the `summarizePipelineStats.R` script

looper/__init__.py

@@ -12,6 +12,7 @@
 from .conductor import SubmissionConductor
 from .pipeline_interface import PipelineInterface
 from .project import Project
+from .sample import Sample
 from ._version import __version__
 from .parser_types import *
 
@@ -24,8 +25,7 @@
     FLAGS, IMPLICATIONS_DECLARATION, SAMPLE_INDEPENDENT_PROJECT_SECTIONS, \
     SAMPLE_NAME_COLNAME
 
-__classes__ = ["PipelineInterface"]
-__all__ = ["Project", "PipelineInterface", "SubmissionConductor"]
+__all__ = ["Project", "PipelineInterface", "Sample", "SubmissionConductor"]
 
 
 GENERIC_PROTOCOL_KEY = "*"
@@ -152,7 +152,7 @@ def add_subparser(cmd):
                 default=DEFAULT_COMPUTE_RESOURCES_NAME,
                 help="YAML file with looper environment compute settings.")
         comp_spec.add_argument(
-                "--compute-packages", dest=COMPUTE_KEY,
+                "--compute-package", dest=COMPUTE_KEY,
                 default=DEFAULT_COMPUTE_RESOURCES_NAME,
                 help="YAML file with looper environment compute settings.")
 

looper/_version.py

@@ -1,2 +1,2 @@
-__version__ = "0.12.0"
+__version__ = "0.12.1"
 

looper/conductor.py

@@ -11,7 +11,8 @@
 from .utils import \
     create_looper_args_text, grab_project_data, fetch_sample_flags
 
-from peppy import Sample, VALID_READ_TYPES
+from .sample import Sample
+from peppy import VALID_READ_TYPES
 
 
 __author__ = "Vince Reuter"
@@ -142,7 +143,7 @@ def add_sample(self, sample, sample_subtype=Sample, rerun=False):
         """
         Add a sample for submission to this conductor.
 
-        :param Sample sample: sample to be included with this conductor's
+        :param peppy.Sample sample: sample to be included with this conductor's
             currently growing collection of command submissions
         :param type sample_subtype: specific subtype associated
             with this new sample; this is used to tailor-make the sample
@@ -155,6 +156,8 @@ def add_sample(self, sample, sample_subtype=Sample, rerun=False):
         :raise TypeError: If sample subtype is provided but does not extend
             the base Sample class, raise a TypeError.
         """
+
+        _LOGGER.debug("Adding {} to conductor for {}".format(sample.name, self.pl_name))
 
         if not issubclass(sample_subtype, Sample):
             raise TypeError("If provided, sample_subtype must extend {}".
@@ -173,19 +176,25 @@ def add_sample(self, sample, sample_subtype=Sample, rerun=False):
                 _LOGGER.info("> Re-running failed sample '%s' for pipeline '%s'.",
                      sample.name, self.pl_name)
                 use_this_sample = True
-            else:
+            if not use_this_sample:
                 _LOGGER.info("> Skipping sample '%s' for pipeline '%s', "
                              "%s found: %s", sample.name, self.pl_name,
                              "flags" if len(flag_files) > 1 else "flag",
                              ", ".join(['{}'.format(
                                  os.path.basename(fp)) for fp in flag_files]))
                 _LOGGER.debug("NO SUBMISSION")
 
-        sample = sample_subtype(sample)
+        if type(sample) != sample_subtype:
+            _LOGGER.debug(
+                "Building {} from {}".format(sample_subtype, type(sample)))
+            sample = sample_subtype(sample.to_dict())
+        else:
+            _LOGGER.debug(
+                "{} is already of type {}".format(sample.name, sample_subtype))
         _LOGGER.debug("Created %s instance: '%s'",
                       sample_subtype.__name__, sample.name)
         sample.prj = grab_project_data(self.prj)
-        
+
         skip_reasons = []
 
         try:

looper/const.py

@@ -36,3 +36,5 @@
 RESULTS_SUBDIR_KEY = "results_subdir"
 SUBMISSION_SUBDIR_KEY = "submission_subdir"
 TEMPLATES_DIRNAME = "jinja_templates"
+IMAGE_EXTS = ('.png', '.jpg', '.jpeg', '.svg', '.gif')
+PROFILE_COLNAMES = ['pid', 'hash', 'cid', 'runtime', 'mem', 'cmd', 'lock']  # this strongly depends on pypiper's profile.tsv format