Merge pull request #124 from refgenie/dev

Release 0.11.0

.github/workflows/black.yml

@@ -1,11 +1,11 @@
 name: Lint
 
-on: [push, pull_request]
+on: [push]
 
 jobs:
   lint:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
-      - uses: psf/black@stable
+      - uses: psf/black@20.8b1

.gitignore

@@ -12,16 +12,16 @@ tests/test/*
 # generic ignore list:
 *.lst
 
-# Compiled source 
+# Compiled source
 *.com
 *.class
 *.dll
 *.exe
 *.o
 *.so
 *.pyc
- 
-# Packages 
+
+# Packages
 # it's better to unpack these files and commit the raw source
 # git has its own built in compression methods
 *.7z
@@ -31,13 +31,13 @@ tests/test/*
 *.rar
 *.tar
 *.zip
- 
-# Logs and databases 
+
+# Logs and databases
 *.log
 *.sql
 *.sqlite
- 
-# OS generated files 
+
+# OS generated files
 .DS_Store
 .DS_Store?
 ._*
@@ -46,7 +46,7 @@ tests/test/*
 ehthumbs.db
 Thumbs.db
 
-# Gedit temporary files 
+# Gedit temporary files
 *~
 
 # libreoffice lock files:

.pre-commit-config.yaml

@@ -0,0 +1,20 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.4.0
+    hooks:
+      - id: trailing-whitespace
+      - id: check-yaml
+      - id: end-of-file-fixer
+      - id: requirements-txt-fixer
+      - id: trailing-whitespace
+
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.7.0
+    hooks:
+      - id: isort
+        args: ["--profile", "black"]
+
+  - repo: https://github.com/psf/black
+    rev: 20.8b1
+    hooks:
+      - id: black

codecov.yml

@@ -1,2 +1,2 @@
 ignore:
-  - "refgenconf/refgenconf_v03.py"
+  - "refgenconf/refgenconf_v03.py"

docs/changelog.md

@@ -1,6 +1,21 @@
 # Changelog
 
-This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format. 
+This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.
+
+## [0.11.0] - 2021-04-27
+### Added
+- `RefGenConf.populate` function to support `refgenie populate`
+- `RefGenConf.populater` function to support `refgenie populater`
+- `RefGenConf.seekr` function to support `refgenie seekr`
+- `refgenconf.looper_refgenie_plugin` -- new looper plugin that populates refgenie paths with looper
+- tag scanning for disallowed characters: `:`, `/`
+
+## Fixed
+- issue with dangling symbolic links in `_refgenie_build` directory; [#122](https://github.com/refgenie/refgenconf/issues/122)
+
+## Changed
+- `RefGenConf.seek` to raise `refgenconf.MissingGenomeError`, rather than `yacman.UndefinedAliasError` if a nonexistent genome is specified
+
 ## [0.10.0] - 2021-03-11
 **After updating to this version your configuration file and genome assets will not be compatible with the software. Please refer to the [upgrade tutorial](config_upgrade_03_to_04.md) for instructions on how to migrate the config between versions.**
 
@@ -14,16 +29,16 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 
 ### Removed
 - `as_string` and `order` option from `listr` method
-	
+
 ### Added
 - `upgrade_config` function for genome configuration file migrating between versions
 - `RefGenConf.compare` method for genome compatibility level determination
 - `as_digests` option in `RefGenConf.listr` method
 - genome config validation on `RefGenConf` object instantiation stage and on every write
-- new progress bar in `RefGenConf.pull`  
+- new progress bar in `RefGenConf.pull`
 - `RefGenConf.get_ta`
 - numerous `RefGenConf` object properties and methods related to genome aliases handling: `genome_aliases`, `genome_aliases_table`, `alias_dir`, `data_dir`, `get_genome_alias`, `get_genome_alias_digest`, `remove_genome_aliases`, `set_genome_alias`, `initialize_genome`. Refer to [API documentation](http://refgenie.databio.org/en/latest/autodoc_build/refgenconf/) for more specific information.
-- `get_asset_table` method, which displays a concise assets table 
+- `get_asset_table` method, which displays a concise assets table
 
 ## [0.9.3] - 2020-09-02
 
@@ -78,7 +93,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 	- `tag_asset` to `tag`
 	- `pull_asset` to `pull`
 	- `get_asset_digest` to `id`
-  	- `get_asset` to `seek`. Moreover the file existence 
+  	- `get_asset` to `seek`. Moreover the file existence
 
 ### Deprecated
 - `get_asset` method. Use `seek` instead
@@ -89,7 +104,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - `genome_archive_config` key to the genome configuration file
 
 ### Changed
-- `genome_archive` key to `genome_archive_folder` in the config file. 
+- `genome_archive` key to `genome_archive_folder` in the config file.
 
 ## [0.6.1] - 2019-12-13
 
@@ -107,11 +122,11 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - `get_asset_digest` method for asset digest retrieval
 - `dir` to the `filepath` method to return an archive enclosing directory
 - `get_asset_digest` method to return the digest for the specified asset
-- `update_genome_servers` method for `genome_servers` attribute manipulation 
+- `update_genome_servers` method for `genome_servers` attribute manipulation
 
 ### Changed
 - `pull_asset` method so that it downloads the archive from a server, makes the object writable, updates it and writes the updates to the refgenie configuration file on disk
-- the way of distribution of `refgenieserver` endpoints operationIds. They are grouped and mapped to a short description   
+- the way of distribution of `refgenieserver` endpoints operationIds. They are grouped and mapped to a short description
 
 ### Fixed
 - overloaded colon in progress bar descriptions; [#73](https://github.com/databio/refgenconf/issues/73)
@@ -172,7 +187,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 
 ### Added
 - `udpate_genomes` method
-- Genome config file version is now verified in `RefGenConf.__init__` 
+- Genome config file version is now verified in `RefGenConf.__init__`
 
 ## [0.2.0] - 2019-06-18
 ### Added

refgenconf/__init__.py

@@ -1,5 +1,25 @@
 from ._version import __version__
-from .refgenconf import RefGenConf, upgrade_config
-from .helpers import select_genome_config, get_dir_digest
 from .const import *
 from .exceptions import *
+from .helpers import *
+from .helpers import get_dir_digest, select_genome_config
+from .populator import looper_refgenie_populate
+from .refgenconf import *
+from .refgenconf import RefGenConf, upgrade_config
+
+__all__ = (
+    [
+        "RefGenConf",
+        "select_genome_config",
+        "get_dir_digest",
+        "GenomeConfigFormatError",
+        "MissingAssetError",
+        "MissingConfigDataError",
+        "MissingGenomeError",
+        "RefgenconfError",
+        "UnboundEnvironmentVariablesError",
+    ]
+    + ["DEFAULT_SERVER"]
+    + CFG_KEY_NAMES
+    + ["looper_refgenie_populate"]
+)

refgenconf/_version.py

@@ -1 +1 @@
-__version__ = "0.10.0"
+__version__ = "0.11.0"

refgenconf/const.py

@@ -10,7 +10,8 @@
 CFG_NAME = "genome configuration"
 CFG_ENV_VARS = ["REFGENIE"]
 CFG_CONST = ["CFG_NAME", "CFG_ENV_VARS"]
-DEFAULT_SERVER = "http://refgenomes.databio.org"
+DEFAULT_SERVER = "http://refgenomes.databio.org"  # "http://rg.databio.org"
+TAG_NAME_BANNED_CHARS = [":", "/"]
 API_VERSION = "v3"
 API_VERSION_2 = "v2"
 DEFAULT_TAG = "default"
@@ -22,6 +23,7 @@
 TEMPLATE_RECIPE_JSON = "build_recipe_{}__{}.json"
 TEMPLATE_TARGET = "{}_{}__{}.flag"
 TEMPLATE_LOG = "build_log_{}__{}.md"
+TEMPLATE_ASSET_DIR_CONTENTS = "asset_dir_contents_{}__{}.json"
 ORI_LOG_NAME = "refgenie_log.md"
 BUILD_STATS_DIR = "_refgenie_build"
 ALIAS_DIR = "alias"
@@ -31,6 +33,7 @@
     "TEMPLATE_RECIPE_JSON",
     "TEMPLATE_TARGET",
     "TEMPLATE_LOG",
+    "TEMPLATE_ASSET_DIR_CONTENTS",
     "ORI_LOG_NAME",
     "BUILD_STATS_DIR",
     "ALIAS_DIR",
@@ -52,9 +55,12 @@
 API_ID_DIGEST = CUSTOM_PFX + "_asset_digest"
 API_ID_RECIPE = CUSTOM_PFX + "_asset_recipe"
 API_ID_LOG = CUSTOM_PFX + "_asset_log"
+API_ID_ASSET_FILE = CUSTOM_PFX + "_asset_file"
+API_ID_ASSET_PATH = CUSTOM_PFX + "_asset_path"
 API_ID_ARCHIVE_DIGEST = CUSTOM_PFX + "_asset_archive_digest"
 API_ID_SPLASH = CUSTOM_PFX + "_asset_splash"
 API_ID_GENOMES_DICT = CUSTOM_PFX + "_genomes_dict"
+API_ID_CONTENTS = CUSTOM_PFX + "_asset_dir_contents"
 
 PRIVATE_API = "_private_api"
 
@@ -76,6 +82,7 @@
         API_VERSION + API_ID_ARCHIVE_DIGEST: "archive digest",
         API_VERSION + API_ID_RECIPE: "build recipe",
         API_VERSION + API_ID_LOG: "build log",
+        API_VERSION + API_ID_CONTENTS: "asset directory contents",
     },
 }
 
@@ -84,6 +91,9 @@
     "API_ID_ARCHIVE",
     "API_ID_DEFAULT_TAG",
     "API_ID_LOG",
+    "API_ID_CONTENTS",
+    "API_ID_ASSET_FILE",
+    "API_ID_ASSET_PATH",
     "API_ID_DIGEST",
     "API_ID_RECIPE",
     "API_ID_ASSET_ATTRS",
@@ -235,6 +245,7 @@
         "CUSTOM_BAR_FMT",
         "API_VERSION",
         "API_VERSION_2",
+        "TAG_NAME_BANNED_CHARS",
         "CONF_STRUCTURE",
         "OPERATION_IDS",
         "CUSTOM_PFX",

refgenconf/exceptions.py

@@ -21,13 +21,13 @@
 
 
 class RefgenconfError(Exception):
-    """ Base exception type for this package """
+    """Base exception type for this package"""
 
     __metaclass__ = abc.ABCMeta
 
 
 class DownloadJsonError(RefgenconfError):
-    """ Non-OK response from a JSON download attempt """
+    """Non-OK response from a JSON download attempt"""
 
     def __init__(self, resp):
         super(DownloadJsonError, self).__init__(
@@ -37,7 +37,7 @@ def __init__(self, resp):
 
 
 class GenomeConfigFormatError(RefgenconfError):
-    """ Exception for invalid genome config file format. """
+    """Exception for invalid genome config file format."""
 
     def __init__(self, msg):
         spacing = " " if msg[-1] in ["?", ".", "\n"] else "; "
@@ -46,55 +46,55 @@ def __init__(self, msg):
 
 
 class MissingAssetError(RefgenconfError):
-    """ Error type for request of an unavailable genome asset. """
+    """Error type for request of an unavailable genome asset."""
 
     pass
 
 
 class MissingTagError(RefgenconfError):
-    """ Error type for request of an unavailable asset tag. """
+    """Error type for request of an unavailable asset tag."""
 
     pass
 
 
 class MissingSeekKeyError(RefgenconfError):
-    """ Error type for request of an unavailable asset seek key. """
+    """Error type for request of an unavailable asset seek key."""
 
     pass
 
 
 class MissingRecipeError(RefgenconfError):
-    """ Error type for request of an unavailable recipe. """
+    """Error type for request of an unavailable recipe."""
 
     pass
 
 
 class MissingConfigDataError(RefgenconfError):
-    """ Missing required configuration instance items """
+    """Missing required configuration instance items"""
 
     pass
 
 
 class ConfigNotCompliantError(GenomeConfigFormatError):
-    """ The format of the config file does not match required version/standards """
+    """The format of the config file does not match required version/standards"""
 
     pass
 
 
 class MissingGenomeError(RefgenconfError):
-    """ Error type for request of unknown genome/assembly. """
+    """Error type for request of unknown genome/assembly."""
 
     pass
 
 
 class UnboundEnvironmentVariablesError(RefgenconfError):
-    """ Use of environment variable that isn't bound to a value. """
+    """Use of environment variable that isn't bound to a value."""
 
     pass
 
 
 class RemoteDigestMismatchError(RefgenconfError):
-    """ Remote digest of the parent asset does not match its local counterpart """
+    """Remote digest of the parent asset does not match its local counterpart"""
 
     def __init__(self, asset, local_digest, remote_digest):
         msg = (