Merge pull request #705 from nsidc/new-populated-places-layer

"Populated places" from "Towns" and "Settlements" layers
nsidc · Aug 2, 2023 · d69ae17 · d69ae17
2 parents 52fc696 + 339646a
commit d69ae17
Show file tree

Hide file tree

Showing 12 changed files with 304 additions and 169 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,9 @@
+# NEXT_VERSION
+
+* Merge "Places/Towns" and "Places/Settlements" layers into one "Populated
+  places" layer. Scale point markers for this layer by population.
+
+
 # v3.0.0alpha4 (2023-07-21)
 
 * QGreenland Custom is no longer supported.

diff --git a/README.md b/README.md
@@ -36,7 +36,7 @@ QGreenland is a data-viewing and analysis platform, and the QGreenland Team does
 not create new data. QGreenland's layers may contain errors from the original
 data providers. QGreenland makes no guarantees about the accuracy and validity
 of data contained in QGreenland. Limited notes on known data issues have been
-added to the 'Towns and settlements' layer metadata. Also, some layers may not
+added to the 'Populated places' layer metadata. Also, some layers may not
 perfectly align with each other due to unidentified georeferencing issues with
 the original data. We recommend using the 'Greenland coastlines 2017' layer as
 the best approximation reference layer for geolocating Greenland's coastline.

diff --git a/doc/_images/OpenFieldcalculator_Larger.png b/doc/_images/OpenFieldcalculator_Larger.png
diff --git a/doc/_images/create_new_field.png b/doc/_images/create_new_field.png
diff --git a/doc/user/tutorials/interacting-with-geospatial-data.md b/doc/user/tutorials/interacting-with-geospatial-data.md
@@ -69,7 +69,7 @@ over the layer name.
 
 **Note**: The QGreenland team has in a few instances included comments on ‘Noted
 Data Issues’.  Read about ‘Noted Data Issues’ in the layer metadata. These are
-currently noted for the ‘Towns and settlements’ layer. Regardless, QGreenland
+currently noted for the ‘Populated places’ layer. Regardless, QGreenland
 makes no guarantees about the accuracy and validity of data contained in
 QGreenland.
 
@@ -214,22 +214,21 @@ Which populated regions in Greenland have more than 5000 people?
 1. Open the **Processing Toolbox** and go to **Vector selection -> Select by attribute**.
 2. Fill in the following parameters:
    -Input layer = Populated places
-   -Selection attribute = population
+   -Selection attribute = Population 2016
    -Operator = ‘>’
    -Value = type in ‘5000’
    -Modify current selection by = creating new selection
    These parameters are telling the program to identify all of the populated places in
    Greenland that have a population greater than 5000.
-3. Click on **Run**. The window will automatically close when processing is
-   complete.
+3. Click on **Run**. Close the window when processing is complete.
 
 There are a couple of ways to view the selected data points, populated places in Greenland
 with more than 5000 people. First, you should see the places that meet this parameter
 highlighted in the map view (make sure the **Populated Places Layer** is toggled on). You can
 also open the Populated places **Layer Attribute Table** and select **Show Selected Features** in
 the bottom left corner. This will hide all records in the **Layer Attribute Table** except for the ones
 you selected, the locations with populations greater than 5000 people.
-If you want to create an entirely new layer based on this feature selection (population>5000),
+If you want to create an entirely new layer based on this feature selection (Population 2016 > 5000),
 you can do so by either 1) right-clicking on the layer you have just selected from and
 choosing **Export -> Save selected features as...**, or by 2) selecting **Extract by attribute**
 under **Vector selection** in the **Processing Toolbox**.
@@ -241,14 +240,15 @@ What is the average size of Greenland’s populated areas?
 1. Open the **Processing Toolbox** and go to **Vector analysis -> Basic statistics for fields**.
 2. Fill in the following parameters:
    -Input layer = Populated places
-   -Field to calculate statistics on = population
+   -Field to calculate statistics on = Population 2016
    -Statistics = Save to temporary file (or whatever your preference is)
 3. Click **Run**.
 
-The **Vector Analysis** window should automatically switch to a view of the Log where you will
-see the results for the population basic statistics (Fig. 8). The value ‘MEAN’ will tell you the
-average size of Greenland’s populated places (1102 people). The value ‘SUM’ will tell you the
-total number of people in Greenland’s metropolitan areas (48,492 people).
+The **Basic statistics for fields** window should automatically switch to a view
+of the Log where you will see the results for the population basic statistics
+(Fig. 8). The value ‘MEAN’ will tell you the average size of Greenland’s
+populated places (702 people). The value ‘SUM’ will tell you the total number
+of people in Greenland’s metropolitan areas (55,494 people).
 
 ![vector_layer_stats](/_images/vector_layer_stats.png)
 Fig. 8: Results of Example 2: Vector Layer Statistics

diff --git a/doc/user/tutorials/video-series-overview.md b/doc/user/tutorials/video-series-overview.md
@@ -58,23 +58,12 @@ If you are not familiar with geographic information systems (GIS) and geospatial
 - Calculate the surface volume area of a raster layer
 
 ```{note}
-In our example, ‘select by attribute’ for a population greater than 5000 would only work
-on an integer field. In v2.0 and later, the population size field is a text string and
-you will first need to use the field calculator to convert the population field into an
-integer. To do so:
+The "Populated places" layer referred to in this video has been updated and moved from "Places/Community map
+(crowd-sourced)" to the root of the "Places/" layer group in QGreenland v3.
 
-1. Open up the attribute table, and click the ‘Open field calculator’ icon.
-
-    ![OpenFieldcalculator_Larger](/_images/OpenFieldcalculator_Larger.png)
-
-2. Check ‘Create a new field’. Designate an output field name (example: pop_numeric) and
-   select the output field type ‘Integer (32 bit)’. In the expression panel, choose the
-   field ‘population’. Click OK.
-
-    ![create_new_field](/_images/create_new_field.png)
+This updated "Populated places" layer has a "Population 2016" attribute that should be used instead of the "population" attribute indicated in the video.
 ```
 
-
 ## [Session 6: Editing Existing and New Layers](https://www.youtube.com/watch?v=98nF6YJAnns&list=PLSRiyMridUCwyu-vqpAFtm8bVERgTvs7q&index=6)
 
 - Edit existing layers in the QGreenland Core Package

diff --git a/qgreenland/ancillary/styles/populated_places.qml b/qgreenland/ancillary/styles/populated_places.qml
diff --git a/qgreenland/config/cfg-lock.json b/qgreenland/config/cfg-lock.json
@@ -81,35 +81,29 @@
           ],
           "id": "buildings"
         },
-        "roads": {
-          "args": [
-            "ogr2ogr",
-            "-oo",
-            "FEATURE_SERVER_PAGING=YES",
-            "{output_dir}/fetched.geojson",
-            "\"https://kort.nunagis.gl/refserver/rest/services/Kortportal/Kortportal_TekniskGrundkort/MapServer/38/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC\""
-          ],
-          "id": "roads"
-        },
-        "settlements": {
+        "populated_places": {
           "args": [
-            "ogr2ogr",
-            "-oo",
-            "FEATURE_SERVER_PAGING=YES",
-            "{output_dir}/fetched.geojson",
-            "\"https://kort.nunagis.gl/refserver/rest/services/Kortportal/Byer_og_bygder/MapServer/0/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC\""
+            "wget",
+            "-O",
+            "{output_dir}/settlements.geojson",
+            "\"https://kort.nunagis.gl/refserver/rest/services/Kortportal/Byer_og_bygder/MapServer/0/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC\"",
+            "&&",
+            "wget",
+            "-O",
+            "{output_dir}/towns.geojson",
+            "\"https://kort.nunagis.gl/refserver/rest/services/Kortportal/Byer_og_bygder/MapServer/1/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC\""
           ],
-          "id": "settlements"
+          "id": "populated_places"
         },
-        "towns": {
+        "roads": {
           "args": [
             "ogr2ogr",
             "-oo",
             "FEATURE_SERVER_PAGING=YES",
             "{output_dir}/fetched.geojson",
-            "\"https://kort.nunagis.gl/refserver/rest/services/Kortportal/Byer_og_bygder/MapServer/1/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC\""
+            "\"https://kort.nunagis.gl/refserver/rest/services/Kortportal/Kortportal_TekniskGrundkort/MapServer/38/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC\""
           ],
-          "id": "towns"
+          "id": "roads"
         }
       },
       "id": "asiaq_nunagis",
@@ -3420,12 +3414,12 @@
         "children": [
           {
             "layer_cfg": {
-              "description": "Points representing towns/cities in Greenland.\n\nQGreenland Team - Noted Data Issues:\n\n* West coast of Greenland, near Paamiut: Ivittuut is abandoned.",
-              "id": "towns",
+              "description": "Points representing populated places in Greenland.\n\nQGreenland Team - Noted Data Issues:\n\n* West coast of Greenland, near Paamiut: Ivittuut is abandoned.\n\n* East Greenland: Ikkatteq is an abandoned airstrip and is not populated.\n\n* Near Ittoqqortoormiit: Uunartoq is abandoned.\n\n* Near Qaanaaq: Qeqertarsuaq and Moriusaq are abandoned.\n\n* Near Upernavik: Tussaaq is abandoned.\n\n* Near Uummannaq: Illorsuit and Nuugaatsiaq are two recently abandoned\n  settlements (2017) due a massive landslide and subsequent tsunami.",
+              "id": "populated_places",
               "in_package": true,
               "input": {
                 "asset": {
-                  "id": "towns"
+                  "id": "populated_places"
                 },
                 "dataset": {
                   "id": "asiaq_nunagis"
@@ -3435,49 +3429,14 @@
               "steps": [
                 {
                   "args": [
-                    "ogr2ogr",
-                    "-lco",
-                    "ENCODING=UTF-8",
-                    "-t_srs",
-                    "EPSG:3413",
-                    "-clipdst",
-                    "{assets_dir}/greenland_rectangle.geojson",
-                    "-makevalid",
-                    "-nln",
-                    "towns",
-                    "-dialect",
-                    "sqlite",
-                    "-sql",
-                    "\"SELECT geometry, Ny_gr\u00f8nlandsk as 'New Greenlandic', Ny_gr\u00f8nlandsk as 'label', Gammel_gr\u00f8nlandsk as 'Old Greenlandic', Dansk as Danish, Alternativt_stednavn  as 'Alternative placename', Indbyggertal_2016 as 'Population 2016' FROM ESRIJSON\"",
-                    "{output_dir}/final.gpkg",
-                    "{input_dir}/fetched.geojson"
+                    "ogrmerge.py",
+                    "-single",
+                    "-o",
+                    "{output_dir}/populated_places.gpkg",
+                    "{input_dir}/*geojson"
                   ],
                   "type": "command"
-                }
-              ],
-              "style": "labeled_point",
-              "tags": [
-                "places"
-              ],
-              "title": "Towns"
-            },
-            "name": "towns"
-          },
-          {
-            "layer_cfg": {
-              "description": "Points representing settlements in Greenland.\n\nQGreenland Team - Noted Data Issues:\n\n* East Greenland: Ikkatteq is an abandoned airstrip and is not populated.\n\n* Near Ittoqqortoormiit: Uunartoq is abandoned.\n\n* Near Qaanaaq: Qeqertarsuaq and Moriusaq are abandoned.\n\n* Near Upernavik: Tussaaq is abandoned.\n\n* Near Uummannaq: Illorsuit and Nuugaatsiaq are two recently abandoned\n  settlements (2017) due a massive landslide and subsequent tsunami.",
-              "id": "settlements",
-              "in_package": true,
-              "input": {
-                "asset": {
-                  "id": "settlements"
                 },
-                "dataset": {
-                  "id": "asiaq_nunagis"
-                }
-              },
-              "show": false,
-              "steps": [
                 {
                   "args": [
                     "ogr2ogr",
@@ -3489,24 +3448,24 @@
                     "{assets_dir}/greenland_rectangle.geojson",
                     "-makevalid",
                     "-nln",
-                    "settlements",
+                    "populated_places",
                     "-dialect",
                     "sqlite",
                     "-sql",
-                    "\"SELECT geometry, Ny_gr\u00f8nlandsk as 'New Greenlandic', Ny_gr\u00f8nlandsk as 'label', Gammel_gr\u00f8nlandsk as 'Old Greenlandic', Dansk as Danish, Alternativt_stednavn  as 'Alternative placename', Indbyggertal_2016 as 'Population 2016' FROM ESRIJSON\"",
+                    "\"SELECT geom, Ny_gr\u00f8nlandsk as 'New Greenlandic', Ny_gr\u00f8nlandsk as 'label', Gammel_gr\u00f8nlandsk as 'Old Greenlandic', Dansk as Danish, Alternativt_stednavn  as 'Alternative placename', Indbyggertal_2016 as 'Population 2016' FROM merged\"",
                     "{output_dir}/final.gpkg",
-                    "{input_dir}/fetched.geojson"
+                    "{input_dir}/populated_places.gpkg"
                   ],
                   "type": "command"
                 }
               ],
-              "style": "labeled_point",
+              "style": "populated_places",
               "tags": [
                 "places"
               ],
-              "title": "Settlements"
+              "title": "Populated places"
             },
-            "name": "settlements"
+            "name": "populated_places"
           },
           {
             "layer_cfg": {
@@ -3632,8 +3591,7 @@
         "settings": {
           "expand": false,
           "order": [
-            ":towns",
-            ":settlements",
+            ":populated_places",
             ":points_of_interest",
             ":roads",
             ":buildings"

diff --git a/qgreenland/config/datasets/asiaq_nunagis.py b/qgreenland/config/datasets/asiaq_nunagis.py
@@ -1,4 +1,5 @@
 from qgreenland.config.helpers.assets.ogr_remote import ogr_remote_asset
+from qgreenland.models.config.asset import CommandAsset
 from qgreenland.models.config.dataset import Dataset
 
 asiaq_nunagis = Dataset(
@@ -15,15 +16,19 @@
             output_file="{output_dir}/fetched.geojson",
             url="https://kort.nunagis.gl/refserver/rest/services/Kortportal/Kortportal_TekniskGrundkort/MapServer/38/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC",
         ),
-        ogr_remote_asset(
-            asset_id="settlements",
-            output_file="{output_dir}/fetched.geojson",
-            url="https://kort.nunagis.gl/refserver/rest/services/Kortportal/Byer_og_bygder/MapServer/0/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC",
-        ),
-        ogr_remote_asset(
-            asset_id="towns",
-            output_file="{output_dir}/fetched.geojson",
-            url="https://kort.nunagis.gl/refserver/rest/services/Kortportal/Byer_og_bygder/MapServer/1/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC",
+        CommandAsset(
+            id="populated_places",
+            args=[
+                "wget",
+                "-O",
+                "{output_dir}/settlements.geojson",
+                '"https://kort.nunagis.gl/refserver/rest/services/Kortportal/Byer_og_bygder/MapServer/0/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC"',
+                "&&",
+                "wget",
+                "-O",
+                "{output_dir}/towns.geojson",
+                '"https://kort.nunagis.gl/refserver/rest/services/Kortportal/Byer_og_bygder/MapServer/1/query/?f=json&where=OBJECTID+is+not+null&outFields=*&orderByFields=OBJECTID+ASC"',
+            ],
         ),
     ],
     metadata={

diff --git a/qgreenland/config/layers/Places/__settings__.py b/qgreenland/config/layers/Places/__settings__.py
@@ -2,8 +2,7 @@
 
 settings = LayerGroupSettings(
     order=[
-        LayerIdentifier("towns"),
-        LayerIdentifier("settlements"),
+        LayerIdentifier("populated_places"),
         LayerIdentifier("points_of_interest"),
         LayerIdentifier("roads"),
         LayerIdentifier("buildings"),

diff --git a/qgreenland/config/layers/Places/populated_places.py b/qgreenland/config/layers/Places/populated_places.py
@@ -0,0 +1,58 @@
+from qgreenland.config.datasets.asiaq_nunagis import asiaq_nunagis
+from qgreenland.config.helpers.steps.ogr2ogr import ogr2ogr
+from qgreenland.config.project import project
+from qgreenland.models.config.layer import Layer, LayerInput
+from qgreenland.models.config.step import CommandStep
+
+populated_places = Layer(
+    id="populated_places",
+    title="Populated places",
+    style="populated_places",
+    description=(
+        """Points representing populated places in Greenland.
+
+QGreenland Team - Noted Data Issues:
+
+* West coast of Greenland, near Paamiut: Ivittuut is abandoned.
+
+* East Greenland: Ikkatteq is an abandoned airstrip and is not populated.
+
+* Near Ittoqqortoormiit: Uunartoq is abandoned.
+
+* Near Qaanaaq: Qeqertarsuaq and Moriusaq are abandoned.
+
+* Near Upernavik: Tussaaq is abandoned.
+
+* Near Uummannaq: Illorsuit and Nuugaatsiaq are two recently abandoned
+  settlements (2017) due a massive landslide and subsequent tsunami."""
+    ),
+    tags=["places"],
+    input=LayerInput(
+        dataset=asiaq_nunagis,
+        asset=asiaq_nunagis.assets["populated_places"],
+    ),
+    steps=[
+        CommandStep(
+            args=[
+                "ogrmerge.py",
+                "-single",
+                "-o",
+                "{output_dir}/populated_places.gpkg",
+                "{input_dir}/*geojson",
+            ],
+        ),
+        *ogr2ogr(
+            input_file="{input_dir}/populated_places.gpkg",
+            output_file="{output_dir}/final.gpkg",
+            boundary_filepath=project.boundaries["data"].filepath,
+            ogr2ogr_args=(
+                "-nln",
+                "populated_places",
+                "-dialect",
+                "sqlite",
+                "-sql",
+                "\"SELECT geom, Ny_grønlandsk as 'New Greenlandic', Ny_grønlandsk as 'label', Gammel_grønlandsk as 'Old Greenlandic', Dansk as Danish, Alternativt_stednavn  as 'Alternative placename', Indbyggertal_2016 as 'Population 2016' FROM merged\"",
+            ),
+        ),
+    ],
+)