WIP: Getting started

USAGE.md

@@ -3,17 +3,26 @@
 * osmconvert64 (https://wiki.openstreetmap.org/wiki/Osmconvert) and osmfilter (http://wiki.openstreetmap.org/wiki/Osmfilter) needs to be found from PATH
 * mgkmap and splitter (http://www.mkgmap.org.uk/) needs to be found from PATH
 * Osmosis for Mapsforge conversion (https://wiki.openstreetmap.org/wiki/Osmosis)
+* wget
+* Python 2 (http://www.python.org)
+  * requests (`pip install requests`)
+  * urllib3 (`pip install urllib3`)
+* java
+  * maven
 * Finnish national grid dataset (only 12km x 12km grid needed)
   * Can be obtained from https://tiedostopalvelu.maanmittauslaitos.fi/geoserver/ows/?service=wfs (License unknown, assumed CC 4.0 like other NLS data)
   * Layer "Grid"
+  (`wget --no-check-certificate -O grid.gml "https://tiedostopalvelu.maanmittauslaitos.fi/geoserver/ows/?service=wfs&request=GetFeature&typeName=Grid"`)
+
 * National topographic database from NLS (about 9.5 GB)
-  * Loaded using helper script `update_data.py`. Simply running `python update_data.py <api_key>` creates necessary directory structure and creates 8 .bat files to load the whole (or just changed grid tiles) topographic database using wget (assumes wget can be found from PATH) 
+  * Loaded using helper script `update_data.py`. Simply running `python update_data.py <api_key>` creates necessary directory structure and creates 8 .bat files to load the whole (or just changed grid tiles) topographic database using wget (assumes wget can be found from PATH).
   * Requires personal API-key to ATOM-feed (http://www.maanmittauslaitos.fi/aineistot-palvelut/latauspalvelut/avoimien-aineistojen-tiedostopalvelu/muutostietopalvelu)
   * Whole process is based on NLS specific GML format because it is the only format containing all of the data and attributes.
 * OpenStreetMap
   * Only features with specific tags are used
   * Load Finland OSM PBF dump from http://download.geofabrik.de/europe/finland-latest.osm.pbf
-  * `join_osm_mtk.bat` creates new o5m file with desired filters applied, and actually merges filtered o5m file to output from the actual topographic database conversion process
+  * `join_osm_mtk.bat` creates new o5m file with desired filters applied, and actually merges filtered o5m file to the output from the actual topographic database conversion process.
+
 * Finnish Transport Agency (Nautical chart data & depth data)
   * Depth data from https://extranet.liikennevirasto.fi/inspirepalvelu/rajoitettu/wfs
     * Whole dataset can be loaded from wfs and is used by the process as a zipped shape
@@ -29,7 +38,12 @@
       * pienriista (not included in styles / tag mapping)
   * National/official hunting areas are not included in the maps because valid concerns about data validity and timeliness.
 
+### OSX / Linux notes
+(So far) The batch files can be run as is provided you mark them executable (e.g. `chmod u+x update_bats/loaddata_0.bat`)
+
 ## MTKtoGarminConverter
+`mvn package` to build ?
+
 This is the actual process doing the conversion from national topographic database GML-files to OSM PBF format and
 combining most of the former datasets with the topograhic data. It does some necessary data transformations and simplifications on the way (defined in `*TagHandler`classes).
 All of the work is done in ETRS-TM35FIN (3067) coordinate system and coordinates are transformed to WGS84 only for PBF output.
@@ -38,7 +52,7 @@ Because of this combination/connection process, the whole conversion process mus
 Process is especially memory optimized and runs consistently well within 2 GB of memory, there is much room for more optimization and for example threaded processing is completely possible.
 
 Process assumes that all data is located in `c:\geodata`. The process works one grid cell at a time and loads auxillary data based on grid cell bounding box and removes unnecessary data from memory after processing. Technically every OGR compatible format should be good for the auxillary data.
-`C:\geodata\mtkgml\` directory is read using 2 level directory structure (`eg. C:\geodata\mtkgml\L4\L44\*.zip`). Processing of the whole topographic database is not necessary, if the directory structure matches. 
+`C:\geodata\mtkgml\` directory is read using 2 level directory structure (`eg. C:\geodata\mtkgml\L4\L44\*.zip`). Processing of the whole topographic database is not necessary, if the directory structure matches.
 
 This step takes about 8 hours with a (slow) SSD, Intel Core i7 920 and 12 GB RAM (and output to HD).
 
@@ -58,7 +72,7 @@ Resulting gmapsupp.img is good to be transferred to GPS or it can be splitted to
 ## Creating Mapsforge map in AWS EC2
 Consult also https://github.com/mapsforge/mapsforge/blob/master/docs/Getting-Started-Map-Writer.md
 
-This steps requires large amounts of memory so EC2 instance is used, r3.2xlarge seems to have enough memory. Tag mapping affects memory usage significantly. 
+This steps requires large amounts of memory so EC2 instance is used, r3.2xlarge seems to have enough memory. Tag mapping affects memory usage significantly.
 
 Styles should be defined at least on some level before the conversion proces because conversion includes only features defined in tag mapping.
 
@@ -73,4 +87,3 @@ The actual process is as follows:
 2. Run Osmosis with writer plugin and tag-mapping and copyright notices with `osmosis --rbf all_osm.osm.pbf --mapfile-writer file=all.map bbox=59.4507573,19.0714057,70.1120744,31.6133108 tag-conf-file=mml_tag-mapping_tidy.xml type=hd comment="(c) NLS, Metsahallitus, Liikennevirasto, OpenStreetMap contributors 2016"`
 
 And out comes all.map containing all of the data in Mapsforge format
-

join_osm_mtk.sh

@@ -0,0 +1,7 @@
+#! /bin/bash
+
+DATA_FOLDER=./geodata/
+
+osmconvert ${DATA_FOLDER}finland-latest.osm.pbf -o=${DATA_FOLDER}finland-latest.o5m
+osmfilter ${DATA_FOLDER}finland-latest.o5m --keep="highway=track =path =footway amenity=shelter =toilets tourism=fireplace =lean_to =wilderness man_made=tower and tower:type=observation" --drop-author --drop-version --verbose --out-o5m >${DATA_FOLDER}finland-filtered.o5m
+osmconvert ${DATA_FOLDER}finland-filtered.o5m --out-o5m | osmconvert - suomi/all.osm.pbf -o=suomi/all_osm.osm.pbf

pom.xml

@@ -22,11 +22,9 @@
       <scope>test</scope>
     </dependency>
     <dependency>
-    	<groupId>org.gdal</groupId>
-    	<artifactId>gdal</artifactId>
-    	<version>2.1.0</version>
-    	<systemPath>C:\Program Files\GDAL\java\gdal.jar</systemPath>
-    	<scope>system</scope>
+      <groupId>org.gdal</groupId>
+      <artifactId>gdal</artifactId>
+      <version>2.1.0</version>
     </dependency>
     <dependency>
     	<groupId>com.google.protobuf</groupId>

update_data.py

@@ -5,6 +5,9 @@
 import datetime
 import StringIO
 
+if len(sys.argv) < 2:
+    print "Usage: python update_data.py API_KEY"
+    sys.exit(1)
 api_key = sys.argv[1]
 
 class MMLGrid(handler.ContentHandler):
@@ -17,9 +20,14 @@ def __init__(self):
         self.files = []
 
         self.nfiles = 8
-
+        BATCH_FILE_DIR = 'update_bats'
+        try:
+            os.makedirs(BATCH_FILE_DIR)
+        except OSError:
+            if not os.path.isdir(BATCH_FILE_DIR):
+                raise
         for i in xrange(self.nfiles):
-            self.files.append(open('update_bats/loaddata_%d.bat' % i,'wb'))
+            self.files.append(open(BATCH_FILE_DIR + '/loaddata_%d.bat' % i,'wb'))
 
         self.gridnum = 0
         self.next = None