Merge pull request #99 from sat-utils/develop

publish 0.3.0

.circleci/config.yml

@@ -44,7 +44,7 @@ jobs:
             pwd
             pip install -r requirements.txt
             pip install -r requirements-dev.txt
-            pytest --cov satsearch test/
+            STAC_API_URL=https://earth-search.aws.element84.com/v0 pytest --cov satsearch test/
       - save_cache:
           key: v1-dependencies-{{ checksum "requirements.txt"}}
           paths:

CHANGELOG.md

@@ -4,11 +4,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [v0.3.0] - 2020-08-21
 
 ## Changed
-- Updated to work with [STAC API v0.9.0-rc1](https://github.com/radiantearth/stac-spec/blob/master/CHANGELOG.md#v090-rc1---2020-01-06)
-- `SATUTILS_API_URL` envvar changed to `STAC_API_URL`
+- Updated to work with STAC API v0.9.0 and v1.0.0-beta.2
+- `SATUTILS_API_URL` envvar changed to `STAC_API_URL` and default value removed. Specify with envvar or pass into Search when using library
+- When downloading, specify `filename_template` for location instead of both `datadir` and `filename`.
+- Update pagination to precisely follow STAC spec
+
+## [v0.2.3] - 2019-06-25
+
+### Changed
+- Default SATUTILS_API_URL changed to account for domain name change
 
 ## [v0.2.2] - 2019-09-20
 
@@ -49,6 +56,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 Initial Release
 
 [Unreleased]: https://github.com/sat-utils/sat-search/compare/master...develop
+[v0.3.0]: https://github.com/sat-utils/sat-search/compare/0.2.3...v0.3.0
+[v0.2.3]: https://github.com/sat-utils/sat-search/compare/0.2.2...v0.2.3
 [v0.2.2]: https://github.com/sat-utils/sat-search/compare/0.2.1...v0.2.2
 [v0.2.1]: https://github.com/sat-utils/sat-search/compare/0.2.0...v0.2.1
 [v0.2.0]: https://github.com/sat-utils/sat-search/compare/0.1.0...v0.2.0

README.md

@@ -2,7 +2,15 @@
 
 [![CircleCI](https://circleci.com/gh/sat-utils/sat-search.svg?style=svg&circle-token=a66861b5cbba7acd4abd7975f804ab061a365e1b)](https://circleci.com/gh/sat-utils/sat-search)
 
-Sat-search is a Python 3 library and a command line tool for discovering and downloading publicly available satellite imagery using a conformant API such as [sat-api](https://github.com/sat-utils/sat-api).
+Sat-search is a Python 3 library and a command line tool for discovering and downloading publicly available satellite imagery using STAC compliant API.
+
+## STAC APIs
+
+Starting with v0.3.0, sat-search does not have a default STAC endpoint. This can be passed as a parameter when using the library, or define the environment variable `STAC_API_URL`. Endpoints known to work are provided in this table:
+
+| Endpoint  | Data |
+| --------  | ----  |
+| https://earth-search.aws.element84.com/v0  | Sentinel-2 |
 
 
 ## Installation
@@ -34,7 +42,7 @@ The table below shows the corresponding versions between sat-search and STAC. Ad
 | --------   | ----  |
 | 0.1.x      | 0.5.x - 0.6.x |
 | 0.2.x      | 0.5.x - 0.7.x |
-| 0.3.x      | 0.9.x |
+| 0.3.x      | 0.9.x - 1.0.0-beta.2 |
 
 
 ## Using sat-search
@@ -52,7 +60,7 @@ The sat-search CLI has an extensive online help that can be printed with the `-h
 $ sat-search -h
 usage: sat-search [-h] {search,load} ...
 
-sat-search (v0.2.0)
+sat-search (v0.3.0)
 
 positional arguments:
   {search,load}
@@ -70,27 +78,31 @@ As can be seen there are two subcommands, `search` and `load`, each of which has
 ```
 $ sat-search search -h
 usage: sat-search search [-h] [--version] [-v VERBOSITY]
-                         [--print-md [PRINTMD [PRINTMD ...]]] [--print-cal]
-                         [--save SAVE] [-c COLLECTION] [--ids [IDS [IDS ...]]]
-                         [--bbox BBOX BBOX BBOX BBOX]
+                         [--print-md [PRINTMD [PRINTMD ...]]]
+                         [--print-cal PRINTCAL] [--save SAVE]
+                         [-c [COLLECTIONS [COLLECTIONS ...]]]
+                         [--ids [IDS [IDS ...]]] [--bbox BBOX BBOX BBOX BBOX]
                          [--intersects INTERSECTS] [--datetime DATETIME]
-                         [--sort [SORT [SORT ...]]] [--found]
-                         [-p [PROPERTY [PROPERTY ...]]] [--url URL]
+                         [-q [QUERY [QUERY ...]]]
+                         [--sortby [SORTBY [SORTBY ...]]] [--found]
+                         [--url URL] [--headers HEADERS] [--limit LIMIT]
 
 optional arguments:
   -h, --help            show this help message and exit
   --version             Print version and exit
   -v VERBOSITY, --verbosity VERBOSITY
-                        0:quiet, 1:error, 2:warning, 3:info, 4:debug (default: 2)
+                        0:quiet, 1:error, 2:warning, 3:info, 4:debug (default:
+                        2)
 
 output options:
   --print-md [PRINTMD [PRINTMD ...]]
-                        Print specified metadata for matched scenes (default: None)
-  --print-cal           Print calendar showing dates (default: False)
+                        Print specified metadata for matched scenes (default:
+                        None)
+  --print-cal PRINTCAL  Print calendar showing dates (default: None)
   --save SAVE           Save results as GeoJSON (default: None)
 
 search options:
-  -c COLLECTION, --collection COLLECTION
+  -c [COLLECTIONS [COLLECTIONS ...]], --collections [COLLECTIONS [COLLECTIONS ...]]
                         Name of collection (default: None)
   --ids [IDS [IDS ...]]
                         One or more scene IDs from provided collection
@@ -102,34 +114,36 @@ search options:
                         GeoJSON Feature (file or string) (default: None)
   --datetime DATETIME   Single date/time or begin and end date/time (e.g.,
                         2017-01-01/2017-02-15) (default: None)
-  --sort [SORT [SORT ...]]
+  -q [QUERY [QUERY ...]], --query [QUERY [QUERY ...]]
+                        Query properties of form KEY=VALUE (<, >, <=, >=, =
+                        supported) (default: None)
+  --sortby [SORTBY [SORTBY ...]]
                         Sort by fields (default: None)
   --found               Only output how many Items found (default: False)
-  -p [PROPERTY [PROPERTY ...]], --property [PROPERTY [PROPERTY ...]]
-                        Properties of form KEY=VALUE (<, >, <=, >=, =
-                        supported) (default: None)
-  --url URL             URL of the API (default: https://n34f767n91.execute-
-                        api.us-east-1.amazonaws.com/prod)
-  --headers HEADERS
-                        JSON Request Headers (file or string) (default: None)
-```
+  --url URL             URL of the API (default: None)
+  --headers HEADERS     Additional request headers (JSON file or string)
+                        (default: None)
+  --limit LIMIT         Limits the total number of items returned (default:
+                        None)
 
 **Search options**
 
-- **collection** - Search only a specific collection. This is a shortcut, collection can also be provided as a property (e.g., `-p "collection=landsat-8-l1"`)
+- **collections** - Search only a specific collections. This is a shortcut, collection can also be provided as a query (e.g., `-q "collection=landsat-8-l1"`)
 - **ids** - Fetch the Item for the provided IDs in the given collection (collection must be provided). All other search options will be ignored.
 - **intersects** - Provide a GeoJSON Feature string or the name of a GeoJSON file containing a single Feature that is a Polygon of an AOI to be searched.
 - **datetime** - Provide a single partial or full datetime (e.g., 2017, 2017-10, 2017-10-11, 2017-10-11T12:00), or two seperated by a slash that defines a range. e.g., 2017-01-01/2017-06-30 will search for scenes acquired in the first 6 months of 2017.
-- **property** - Allows searching for any other scene properties by providing the pair as a string (e.g. `-p "landsat:row=42"`, `-p "eo:cloud_cover<10"`). Supported symbols include: =, <, >, >=, and <=
-- **sort** - Sort by specific properties in ascending or descending order. A list of properties can be provided which will be used for sorting in that order of preference. By default a property will be sorted in descending order. To specify the order the property can be preceded with '<' (ascending) or '>' (descending). e.g., `--sort ">datetime" "<eo:cloud_cover" will sort by descending date, then by ascending cloud cover
-- **found** - This will print out the total number of scenes found, then exit without fetching the actual items. 
-- **url** - The URL endpoint of a STAC compliant API, this can also be set with the environment variable SATUTILS_API_URL
+- **query** - Allows searching for any other scene properties by providing the pair as a string (e.g. `-p "landsat:row=42"`, `-p "eo:cloud_cover<10"`). Supported symbols include: =, <, >, >=, and <=
+- **sortby** - Sort by specific properties in ascending or descending order. A list of properties can be provided which will be used for sorting in that order of preference. By default a property will be sorted in descending order. To specify the order the property can be preceded with '<' (ascending) or '>' (descending). e.g., `--sort ">datetime" "<eo:cloud_cover" will sort by descending date, then by ascending cloud cover
+- **found** - This will print out the total number of scenes found, then exit without fetching the actual items (i.e., the query is made with limit=0). 
+- **url** - The URL endpoint of a STAC compliant API, this can also be set with the environment variable STAC_API_URL
+- **headers** - Additional request headers useful for specifying authentication parameters
+- **limit** - Limits total number of Items returned
 
 **Output options**
 These options control what to do with the search results, multiple switches can be provided.
 
 - **print-md** - Prints a list of specific metadata fields for all the scenes. If given without any arguments it will print a list of the dates and scene IDs. Otherwise it will print a list of fields that are provided. (e.g., --print-md date eo:cloud_cover eo:platform will print a list of date, cloud cover, and the satellite platform such as WORLDVIEW03)
-- **print-cal** - Prints a text calendar (see iumage below) with specific days colored depending on the platform of the scene (e.g. landsat-8), along with a legend.
+- **print-cal** - Prints a text calendar (see image below) with specific days colored grouped by a provided property name (e.g. platform), along with a legend.
 - **save** - Saves results as a FeatureCollection. The FeatureCollection 'properties' contains all of the arguments used in the search and the 'features' contain all of the individual scenes, with individual scene metadata merged with collection level metadata (metadata fields that are the same across all one collection, such as eo:platform)
 
 ![](images/calendar.png)
@@ -140,11 +154,9 @@ These options control what to do with the search results, multiple switches can
 Scenes that were previously saved with `sat-search search --save ...` can be loaded with the `load` subcommand.
 
 ```
-$ sat-search load -h
-usage: sat-search load [-h] [--version] [-v VERBOSITY]
-                       [--print-md [PRINTMD [PRINTMD ...]]] [--print-cal]
-                       [--save SAVE] [--datadir DATADIR] [--filename FILENAME]
-                       [--download [DOWNLOAD [DOWNLOAD ...]]]
+usage: sat-search load [-h] [--version] [-v VERBOSITY] [--print-md [PRINTMD [PRINTMD ...]]] [--print-cal PRINTCAL]
+                       [--save SAVE] [--filename_template FILENAME_TEMPLATE]
+                       [--download [DOWNLOAD [DOWNLOAD ...]]] [--requester-pays]
                        items
 
 positional arguments:
@@ -154,23 +166,21 @@ optional arguments:
   -h, --help            show this help message and exit
   --version             Print version and exit
   -v VERBOSITY, --verbosity VERBOSITY
-                        0:quiet, 1:error, 2:warning, 3:info, 4:debug (default:
-                        2)
+                        0:quiet, 1:error, 2:warning, 3:info, 4:debug (default: 2)
 
 output options:
   --print-md [PRINTMD [PRINTMD ...]]
-                        Print specified metadata for matched scenes (default:
-                        None)
-  --print-cal           Print calendar showing dates (default: False)
+                        Print specified metadata for matched scenes (default: None)
+  --print-cal PRINTCAL  Print calendar showing dates (default: None)
   --save SAVE           Save results as GeoJSON (default: None)
 
 download options:
-  --datadir DATADIR     Directory pattern to save assets (default:
-                        ./${eo:platform}/${date})
-  --filename FILENAME   Save assets with this filename pattern based on
-                        metadata keys (default: ${id})
+  --filename_template FILENAME_TEMPLATE
+                        Save assets with this filename pattern based on metadata keys (default:
+                        ${collection}/${date}/${id})
   --download [DOWNLOAD [DOWNLOAD ...]]
                         Download assets (default: None)
+  --requester-pays      Acknowledge paying egress costs for downloads (if in request pays bucket) (default: False)
 ```
 
 Note that while the search options are gone, output options are still available and can be used with the search results loaded from the file. There is also a new series of options for downloading data.
@@ -180,18 +190,16 @@ When loading results from a file, the user now has the option to download assets
 
 **Download options**
 These control the downloading of assets. Both datadir and filename can include metadata patterns that will be substituted per scene.
-- **datadir** - This specifies where downloaded assets will be saved to. It can also be specified by setting the environment variable SATUTILS_DATADIR.
-- **filename** - The name of the file to save. It can also be set by setting the environment variable SATUTILS_FILENAME
 - **download** - Provide a list of keys to download these assets. More information on downloading data is provided below.
-
-**Metadata patterns**
-Metadata patterns can be used in **datadir** and **filename** in order to have custom path and filenames based on the Item metadata. For instance specifying datadir as "./${eo:platform}/${date}" will save assets for each Item under directories of the platform and the date. So a landsat-8 Item from June 20, 2018 will have it's assets saved in a directory './landsat-8/2017-06-20'. For filenames these work exactly the same way, except the filename will contain a suffix containing the asset key and the appropriate extension.
-
-```
-    sat-search load scenes.json --download thumbnail MTL
-```
-
-In this case the defaults for `datadir` ("./${eo:platform}/${date}") and `filename` ("${id}") are used so the download files are saved like this:
+- **filename_template** - This specifies the filename prefix where downloaded assets will be saved to based on a template using properties from the specific STAC Item. Supported fields:
+    - ${id}: The ID of the STAC Item
+    - ${collection}: The collection of the STAC Item
+    - ${date}: The date portion of the `datetime` property
+    - ${year}: The year of the `datetime` property
+    - ${month}: The month of the `datetime` property
+    - ${day}: The day of the month of the `datetime` property
+    - ${<property>}: Any STAC Item property may be used, e.g. "${eo:cloud_cover}", "${platform}
+  The actual filename will be this prefix followed by the asset key and an appropriate extension. For example, specifying `filename_template` as "./${eo:platform}/${date}/${id}" will save assets for each Item under directories of the platform and the date. Thus, a landsat-8 Item from June 20, 2018 will have it's assets saved in a directory './landsat-8/2017-06-20/'. A metadata asset with the key `MTL` would be saved as './landsat-8/2017-06-20/LC80090292018275LGN00_MTL.TIF'. The last component of the filename_template is taken as the filename. See example directory structure below.
 
 ```
 landsat-8/
@@ -212,4 +220,4 @@ A shortcut to download all of the assets is available by providing "ALL" as the
 This [Jupyter notebook tutorial](tutorial-1.ipynb) covers all the main features of the library.
 
 ## About
-sat-search was created by [Development Seed](<http://developmentseed.org>) and is part of a collection of tools called [sat-utils](https://github.com/sat-utils).
+sat-search is part of a collection of tools called [sat-utils](https://github.com/sat-utils).

satsearch/__init__.py

@@ -1,4 +1,4 @@
-from satsearch.search import Search, API_URL
+from satsearch.search import Search
 from satsearch.version import __version__
 
 import logging

satsearch/cli.py

@@ -5,10 +5,12 @@
 import sys
 
 from .version import __version__
-from satsearch import Search, API_URL
+from satsearch import Search
 from satstac import ItemCollection
 from satstac.utils import dict_merge
 
+API_URL = os.getenv('STAC_API_URL', None)
+
 
 class SatUtilsParser(argparse.ArgumentParser):
 
@@ -27,7 +29,7 @@ def __init__(self, *args, **kwargs):
         self.download_group.add_argument('--filename_template', default='${collection}/${date}/${id}',
                            help='Save assets with this filename pattern based on metadata keys')
         self.download_group.add_argument('--download', help='Download assets', default=None, nargs='*')
-        h = 'Acknowledge paying egress costs for downloads (if in request pays bucket)'
+        h = 'Acknowledge paying egress costs for downloads (if in requester pays bucket on AWS)'
         self.download_group.add_argument('--requester-pays', help=h, default=False, action='store_true', dest='requester_pays')
 
         self.output_parser = argparse.ArgumentParser(add_help=False)
@@ -36,7 +38,6 @@ def __init__(self, *args, **kwargs):
         self.output_group.add_argument('--print-md', help=h, default=None, nargs='*', dest='printmd')
         h = 'Print calendar showing dates'
         self.output_group.add_argument('--print-cal', help=h, dest='printcal')
-        self.output_group.add_argument('--print-assets', help=h, dest='printassets', default=False, action='store_true')
         self.output_group.add_argument('--save', help='Save results as GeoJSON', default=None)
 
     def parse_args(self, *args, **kwargs):
@@ -98,6 +99,7 @@ def newbie(cls, *args, **kwargs):
         parser.search_group.add_argument('--found', help=h, action='store_true', default=False)
         parser.search_group.add_argument('--url', help='URL of the API', default=API_URL)
         parser.search_group.add_argument('--headers', help='Additional request headers (JSON file or string)', default=None)
+        parser.search_group.add_argument('--limit', help='Limits the total number of items returned', default=None)
 
         parents.append(parser.download_parser)
         lparser = subparser.add_parser('load', help='Load items from previous search', parents=parents)
@@ -112,18 +114,19 @@ def __call__(self, parser, namespace, values, option_string=None):
                 setattr(namespace, n, {'eq': v})
 
 
-def main(items=None, printmd=None, printcal=None, printassets=None,
+def main(items=None, printmd=None, printcal=None,
          found=False, filename_template='${collection}/${date}/${id}',
          save=None, download=None, requester_pays=False, headers=None, **kwargs):
     """ Main function for performing a search """
 
     if items is None:
         ## if there are no items then perform a search
         search = Search.search(headers=headers, **kwargs)
+        ## Commenting out found logic until functions correctly.
         if found:
-            num = search.found(headers=headers)
-            print('%s items found' % num)
-            return num
+             num = search.found(headers=headers)
+             print('%s items found' % num)
+             return num
         items = search.items(headers=headers)
     else:
         # otherwise, load a search from a file
@@ -139,9 +142,6 @@ def main(items=None, printmd=None, printcal=None, printassets=None,
     if printcal:
         print(items.calendar(printcal))
 
-    if printassets:
-        print(items.assets_definition())
-
     # save all metadata in JSON file
     if save is not None:
         items.save(filename=save)