Merge pull request #432 from mjohns-databricks/docs-20230915

Docs 20230915

README.md

@@ -45,13 +45,13 @@ Image1: Mosaic logical design.
 We recommend using Databricks Runtime versions 11.3 LTS or 12.2 LTS with Photon enabled; this will leverage the 
 Databricks H3 expressions when using H3 grid system. 
 
-:warning: **Mosaic 0.3 series does not support DBR 13** (coming soon with Mosaic 0.4 series); also, DBR 10 is no longer supported in Mosaic. 
+:warning: **Mosaic 0.3 series does not support DBR 13** (coming soon); also, DBR 10 is no longer supported in Mosaic. 
 
 As of the 0.3.11 release, Mosaic issues the following warning when initialized on a cluster that is neither Photon Runtime nor Databricks Runtime ML [[ADB](https://learn.microsoft.com/en-us/azure/databricks/runtime/) | [AWS](https://docs.databricks.com/runtime/index.html) | [GCP](https://docs.gcp.databricks.com/runtime/index.html)]:
 
-> DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster from version v0.4.0+. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits).
+> DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster after v0.3.x. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits).
 
-If you are receiving this warning in v0.3.11, you will want to change to a supported runtime prior to updating Mosaic to run 0.4.0. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider.
+If you are receiving this warning in v0.3.11+, you will want to begin to plan for a supported runtime. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider.
 
 ### Documentation
 

docs/source/api/spatial-functions.rst

@@ -459,7 +459,7 @@ st_distance
 
 .. function:: st_distance(geom1, geom2)
 
-    Compute the distance between `geom1` and `geom2`.
+    Compute the euclidean distance between `geom1` and `geom2`.
 
     :param geom1: Geometry
     :type geom1: Column
@@ -509,7 +509,7 @@ st_distance
     |      15.652475842498529|
     +------------------------+
 
-.. note:: Results of this function are always expressed in the original units of the input geometries.
+.. note:: Results of this euclidean distance function are always expressed in the original units of the input geometries, e.g. for WGS84 (SRID 4326) units are degrees.
 
 st_dump
 *******
@@ -744,7 +744,7 @@ st_haversine
     |                   10007.55722101796|
     +------------------------------------+
 
-.. note:: Results of this function are always expressed in km^2, while the input lat/lng pairs are expected to be in degrees.
+.. note:: Results of this function are always expressed in km, while the input lat/lng pairs are expected to be in degrees. The radius used (in km) is 6371.0088.
 
 
 st_hasvalidcoordinates

docs/source/index.rst

@@ -42,14 +42,14 @@
 Mosaic is an extension to the `Apache Spark <https://spark.apache.org/>`_ framework that allows easy and fast processing of very large geospatial datasets.
 
 .. warning::
-    From version 0.4.0, Mosaic will require either
+    From versions after 0.3.x, Mosaic will require either
      * Databricks Runtime 11.2+ with Photon enabled
      * Databricks Runtime for ML 11.2+
 
-    Mosaic 0.3 series does not support DBR 13 (coming soon with Mosaic 0.4 series); 
+    Mosaic 0.3 series does not yet support DBR 13 (coming soon); 
     also, DBR 10 is no longer supported in Mosaic. 
 
-We recommend using Databricks Runtime versions 11.3 LTS or 12.2 LTS with Photon enabled; 
+We currently recommend using Databricks Runtime versions 11.3 LTS or 12.2 LTS with Photon enabled; 
 this will leverage the Databricks H3 expressions when using H3 grid system. 
 
 Mosaic provides:

docs/source/usage/installation.rst

@@ -6,22 +6,22 @@ Supported platforms
 ###################
 
 .. warning::
-    From version 0.4.0, Mosaic will require either
+    From versions after 0.3.x, Mosaic will require either
      * Databricks Runtime 11.2+ with Photon enabled
      * Databricks Runtime for ML 11.2+
 
-    Mosaic 0.3 series does not support DBR 13 (coming soon with Mosaic 0.4 series); 
+    Mosaic 0.3 series does not support DBR 13 (coming soon); 
     also, DBR 10 is no longer supported in Mosaic. 
 
 We recommend using Databricks Runtime versions 11.3 LTS or 12.2 LTS with Photon enabled; 
 this will leverage the Databricks H3 expressions when using H3 grid system.  
 As of the 0.3.11 release, Mosaic issues the following warning when initialized on a cluster
 that is neither Photon Runtime nor Databricks Runtime ML [`ADB <https://learn.microsoft.com/en-us/azure/databricks/runtime/>`__ | `AWS <https://docs.databricks.com/runtime/index.html>`__ | `GCP <https://docs.gcp.databricks.com/runtime/index.html>`__]:
 
-    DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster from version v0.4.0+. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits).
+    DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster after v0.3.x. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits).
 
-If you are receiving this warning in v0.3.11+, you will want to change to a supported runtime prior 
-to updating Mosaic to run 0.4.0. The reason we are making this change is that we are streamlining Mosaic
+If you are receiving this warning in v0.3.11+, you will want to begin to plan for a supported runtime.
+The reason we are making this change is that we are streamlining Mosaic
 internals to be more aligned with future product APIs which are powered by Photon. Along this direction 
 of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider.
 

modules/python/gdal_package/README.md

@@ -2,7 +2,7 @@
 
 > Current version is 3.4.3 (to match GDAL).
 
-This is a filetree (vs apt based) drop-in packaging of GDAL with Java Bindings for Ubuntu 20.04 (Focal Fossa) which is used by [Databricks Runtime](https://docs.databricks.com/release-notes/runtime/releases.html) (DBR) 11+. 
+This is a filetree (vs apt based) drop-in packaging of GDAL with Java Bindings for Ubuntu 20.04 (Focal Fossa) which is used by [Databricks Runtime](https://docs.databricks.com/release-notes/runtime/releases.html) (DBR) 11 and 12 (not DBR 13 which is Ubuntu 22.04). 
 
 1. `gdal-3.4.3-filetree.tar.xz` is ~50MB - it is extracted with `tar -xf gdal-3.4.3-filetree.tar.xz -C /`
 2. `gdal-3.4.3.-symlinks.tar.xz` is ~19MB - it is extracted with `tar -xhf gdal-3.4.3-symlinks.tar.xz -C /`
@@ -14,4 +14,3 @@ An [init script](https://docs.databricks.com/clusters/init-scripts.html) is prov
 * This is a very specific packaging for GDAL + dependencies which removes any libraries that are already provided by DBR, so it will not be not useful outside Databricks.
 * It additionally includes GDAL shared objects (`.so`) for Java Bindings, GDAL 3.4.3 Python bindings, and tweak for OSGEO as currently supplied by [UbuntuGIS PPA](https://launchpad.net/~ubuntugis/+archive/ubuntu/ubuntugis-unstable) based init script [install-gdal-databricks.sh](https://github.com/databrickslabs/mosaic/blob/main/src/main/resources/scripts/install-gdal-databricks.sh) provided by Mosaic. This install replaces the existing way on Mosaic, so choose one or the other.
 * The GDAL JAR for 3.4 is not included but is provided by Mosaic itself and added to your Databricks cluster as part of the [enable_gdal](https://databrickslabs.github.io/mosaic/usage/install-gdal.html#enable-gdal-for-a-notebook) called when configuring Mosaic for GDAL. Separately, the JAR could be added as a [cluster-installed library](https://docs.databricks.com/libraries/cluster-libraries.html#cluster-installed-library), e.g. through Maven coordinates `org.gdal:gdal:3.4.0` from [mvnrepository](https://mvnrepository.com/artifact/org.gdal/gdal/3.4.0).
-* Mosaic will soon be able to directly leverage this [PyPI](https://pypi.org/project/databricks-mosaic-gdal/) project and be able to altogether avoid the init script as a precursor to calling [enable_gdal](https://databrickslabs.github.io/mosaic/usage/install-gdal.html#enable-gdal-for-a-notebook). So check Mosaic [GDAL Installation Guide](https://databrickslabs.github.io/mosaic/usage/install-gdal.html#) for any changes on/around APR 2023.

modules/python/gdal_package/databricks-mosaic-gdal/resources/scripts/mosaic-gdal-3.4.3-filetree-init.sh

@@ -11,7 +11,7 @@
 #  2. This will unpack files directly into the filetree across cluster nodes (vs run apt install)
 #
 # -- install databricks-mosaic-gdal on cluster 
-# - from pypi.org (once available)
+# - use version 3.4.3 (exactly) from pypi.org
 pip install databricks-mosaic-gdal==3.4.3
 
 # -- find the install dir

src/main/scala/com/databricks/labs/mosaic/functions/MosaicContext.scala

@@ -944,10 +944,10 @@ object MosaicContext extends Logging {
         if (!isML && !isPhoton) {
             // Print out the warnings both to the log and to the console
             logWarning("DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime")
-            logWarning("DEPRECATION WARNING: Mosaic will stop working on this cluster from version v0.4.0+.")
+            logWarning("DEPRECATION WARNING: Mosaic will stop working on this cluster after v0.3.x.") 
             logWarning("Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits).")
             println("DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime")
-            println("DEPRECATION WARNING: Mosaic will stop working on this cluster from version v0.4.0+.")
+            println("DEPRECATION WARNING: Mosaic will stop working on this cluster after v0.3.x.")
             println("Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits).")
             false
         } else {