national-dataset-query

national-dataset-query is a command-line tool that extracts data from ESRI REST endpoints when nothing else will.

Table of Contents

• Installation
• Usage
• Options
• Testing
• Formats
  • Input
    • ESRI Protobuf
    • ESRI JSON
  • Output
    • GeoJSON
    • GeoJSONSeq
    • GeoPackage

Installation

To install national-dataset-query, you need to have Node.js installed.

You will then need to build it:

git clone https://github.com/nationalparkservice/national-dataset-query.git
cd ./national-dataset-query
npm run build

Usage

To use national-dataset-query, run the following command in your terminal:

nps run start -- --url <URL>

The <URL> should be the URL of the ESRI REST endpoint, for example https://sampleserver6.arcgisonline.com/arcgis/rest/services/LocalGovernment/Recreation/FeatureServer/2.

By default, the output is printed to the console in GeoJSON format. You can specify other options using flags, as shown in the Options section.

Options

Flag	Description
-h, --help	Display this usage guide.
-u, --url	The URL of the ESRI Rest Endpoint. MapServer or Feature Server (ex. https://.../FeatureServer/0)
-w, --where string	ESRI Style Where (Defaults to 1=1)
-f, --format string	[gpkg, geojson, geojsonseq]
-o, --output string	The file to write out (if set, type becomes file)
-y, --pretty	Pretty Print JSON (geojsonseq will override this)
-c, --feature-count num	Features per query, reduce this number if you're seeing a lot of bad requests from the server (Default is server default)
-j, --json	Use ESRI json to download data (otherwise it will try to use the esri protobuf format)
-p, --progress	Show progress during the process
-l, --layer-name	For GPKG files, specifies the layer-name, if unset, it will use the filename
-b, --no-bbox	Does not calculate a bbox for each feature. (Bboxs are slower to generate, but may speed up calculations on the resulting file)

Examples

Simplest GeoJSON Example

npm run start -- --url "https://sampleserver6.arcgisonline.com/arcgis/rest/services/LocalGovernment/Recreation/FeatureServer/2"

to GeoJSONSeq File Example

GeoJSONSeq allows parallel processing in Tippecanoe

npm run start -- \
--url "https://sampleserver6.arcgisonline.com/arcgis/rest/services/LocalGovernment/Recreation/FeatureServer/2" \
--format geojsonseq \
--output ./example.geojsonseq

to GeoPackage File Example

GeoPackages load into PostgreSQL much faster than GeoJSON or GeoJSONSeq

npm run start -- \
--url "https://sampleserver6.arcgisonline.com/arcgis/rest/services/LocalGovernment/Recreation/FeatureServer/2" \
--format gpkg \
--output ./example.gpkg

Testing

To test national-dataset-query, run the following command in your terminal:

npm run test

Formats

Input

ESRI Protobuf (PBF)

No, this isn't the same as MapBox Vector Tiles Protobuf schema, although both use the same underlying Protobuf format.

ESRI Protobuf format provides "zig-zag encoded" points on a quantized grid that are stored in a binary format (Google Protobuf). (You can read all about quantization parameters). The format is very similar to the ESRI JSON format, which is why this library uses the Terraformer JS library under the hood to convert the Protobuf data to GeoJSON.

This format is much faster than the ESRI JSON format, but is not compatible with all ArcGSI REST Servers/

ESRI JSON

ESRI has their own spatial format that provides some more information than standard GeoJSON. All ESRI ArcGIS REST Vector Endpoints support from form of ESRI JSON, so when PBF is not supported or a query is run with --json, this is the source format that is used. You can find more information about the ESRI JSON format in the ArcGIS REST API Documentation.

Output

GeoJSON

GeoJSON is a standard GeoSpatial format that has the most interoperability. This is the "native" geospatial format used by national-dataset-query, so all projection conversions are done on the server.

GeoJSONSeq

This is a format creates individual GeoJSON Features into a format that is more useful for parallel processing. The features are separated by a newline (LF), which makes it Newline Delimited JSON. It is useful for Tippecanoe.

You can read more abot the format on the ogr2ogr page. There is also RS delimited version, but it's not supported by this tool since I don't have a use for it, but if you do, open an issue.

GeoPackage

GeoPackages are sqlite files that follow a standard for spatial data storage. In national-dataset-query, these GeoPackage files are created with the better-sqlite3 library, they do not use spatialite, and do not have a spatial index. GeoPackage uses the Well-Known-Binary format, so these files are very performant for importing to PostGIS (which uses its own version of WKB).

Since the output GeoPackages aren't spatialite files or spatially indexed, they may be a little slower in tools like QGIS. You can use a tool like ogr2ogr or QGIS to convert this GeoPackage to one with a spatial index if needed.