Skip to content

Commit

Permalink
Add support for Spatialite and MySQL WKB dialects (#153)
Browse files Browse the repository at this point in the history 
This PR adds support for
[Spatialite](https://www.gaia-gis.it/gaia-sins/BLOB-Geometry.html) and
[MySQL](https://dev.mysql.com/doc/refman/8.0/en/gis-data-formats.html)
WKB dialects.

I'd happily discuss the implementation. Quirks of the spatialite dialect
that required some refactoring:
- A geometry blob ends with `0xFE`. In order to know wether the
`polygon_end` (or alike) is the last one, we keep track of the
collection entity nesting level. If the nesting level is 0 when a
geometry ends, it means it is the last one of the blob and the
terminating byte `0xFE` must be appended. Even though `MULTI*`
geometries cannot be part of a `GEOMETRYCOLLECTION` in Spatialite at the
moment, this was my easiest go at the problem - and it also makes it
more future-proof in case that feature is added at some point.
- In contrary to EWKB, the byte order and SRID info are not repeated in
the nested entities headers, which means they have to be inherited from
the parent entity info. In order to do that, I had to change the
signature of the `read_header` parameter of `process_wkb_geom_n` so that
the parent geometry info can be passed down to its children.
- The "compressed" polygon and linestring geometries stores their X,Y,Z
coordinates as `f32` values, relative to the previous one (with the
exception of the first and the last one), which required some
refactoring to `process_coord` to avoid code duplication between the two
implementations.

Side notes:
- Shouldn't `WkbWriter` implement `GeomProcessor::srid()` ?
- Only read support has been implemented for Compressed and TinyPoint
geometries, write isn't implemented.
- Spatialite header stores the envelope as `[minx, miny, maxx, maxy]`
(like GeoJSON) whereas gpkg stores it as `[minx, maxx, miny, maxy]`

I also refactored the roundtrip tests so they can be shared by the 5
dialects.

---------

Co-authored-by: Aurèle Nitoref <aurele.nitoref@icloud.com>
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
  • Loading branch information
@Oreilles @nyurik
3 people authored Aug 12, 2023
1 parent 5e92027 commit a92ef80
Show file tree
Hide file tree
Showing 7 changed files with 662 additions and 161 deletions.
4 changes: 2 additions & 2 deletions geozero/src/arrow/geoarrow_reader.rs
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use crate::error::Result;
use crate::wkb::wkb_reader::{process_wkb_geom_n, read_wkb_header};
use crate::wkb::wkb_reader::{process_wkb_geom_n, read_wkb_header, read_wkb_nested_header};
use crate::{GeomProcessor, GeozeroGeometry};
use arrow2::array::BinaryArray;
use arrow2::types::Offset;
Expand All @@ -26,7 +26,7 @@ pub fn process_geoarrow_wkb_geom<T: Offset>(
for i in 0..array_len {
let raw = &mut array.value(i);
let info = read_wkb_header(raw)?;
process_wkb_geom_n(raw, &info, read_wkb_header, i, processor)?;
process_wkb_geom_n(raw, &info, read_wkb_nested_header, i, processor)?;
}

processor.geometrycollection_end(array_len - 1)
Expand Down
2 changes: 2 additions & 0 deletions geozero/src/error.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ pub enum GeozeroError {
// GeometryProcessor
#[error("accessing requested coordinate")]
Coord,
#[error("invalid SRID value `{0}`")]
Srid(i32),
#[error("processing geometry `{0}`")]
Geometry(String),
// General
Expand Down
26 changes: 13 additions & 13 deletions geozero/src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,19 @@
//!
//! ## Format conversion overview
//!
//! | | [`GeozeroGeometry`] | Dimensions | [`GeozeroDatasource`] | Geometry Conversion | [`GeomProcessor`] |
//! |-----------|----------------------------------------------------------------------|------------|----------------------------------------------------------------------|---------------------|-----------------------------------------|
//! | CSV | [csv::Csv], [csv::CsvString] | XY | - | [ProcessToCsv] | [CsvWriter](csv::CsvWriter) |
//! | GDAL | `gdal::vector::Geometry` | XYZ | - | [ToGdal] | [GdalWriter](gdal::GdalWriter) |
//! | geo-types | `geo_types::Geometry<f64>` | XY | - | [ToGeo] | [GeoWriter](geo_types::GeoWriter) |
//! | GeoArrow | `arrow2::array::BinaryArray` | XY | - | - | - |
//! | GeoJSON | [GeoJson](geojson::GeoJson), [GeoJsonString](geojson::GeoJsonString) | XYZ | [GeoJsonReader](geojson::GeoJsonReader), [GeoJson](geojson::GeoJson) | [ToJson] | [GeoJsonWriter](geojson::GeoJsonWriter) |
//! | GEOS | `geos::Geometry` | XYZ | - | [ToGeos] | [GeosWriter](geos::GeosWriter) |
//! | GPX | | XY | [GpxReader](gpx::GpxReader) | | |
//! | MVT | [mvt::tile::Feature] | XY | [mvt::tile::Layer] | [ToMvt] | [MvtWriter](mvt::MvtWriter) |
//! | SVG | - | XY | - | [ToSvg] | [SvgWriter](svg::SvgWriter) |
//! | WKB | [Wkb](wkb::Wkb), [Ewkb](wkb::Ewkb), [GpkgWkb](wkb::GpkgWkb) | XYZM | - | [ToWkb] | [WkbWriter](wkb::WkbWriter) |
//! | WKT | [wkt::WktStr], [wkt::WktString], [wkt::EwktStr], [wkt::EwktString] | XYZM | [wkt::WktReader], [wkt::WktStr], [wkt::WktString] | [ToWkt] | [WktWriter](wkt::WktWriter) |
//! | | [`GeozeroGeometry`] | Dimensions | [`GeozeroDatasource`] | Geometry Conversion | [`GeomProcessor`] |
//! |-----------|--------------------------------------------------------------------------------------------------------------------------|------------|--------------------------------------------------------------------------------------|---------------------|-----------------------------------------|
//! | CSV | [csv::Csv], [csv::CsvString] | XY | - | [ProcessToCsv] | [CsvWriter](csv::CsvWriter) |
//! | GDAL | `gdal::vector::Geometry` | XYZ | - | [ToGdal] | [GdalWriter](gdal::GdalWriter) |
//! | geo-types | `geo_types::Geometry<f64>` | XY | - | [ToGeo] | [GeoWriter](geo_types::GeoWriter) |
//! | GeoArrow | `arrow2::array::BinaryArray` | XY | - | - | - |
//! | GeoJSON | [GeoJson](geojson::GeoJson), [GeoJsonString](geojson::GeoJsonString) | XYZ | [GeoJsonReader](geojson::GeoJsonReader), [GeoJson](geojson::GeoJson) | [ToJson] | [GeoJsonWriter](geojson::GeoJsonWriter) |
//! | GEOS | `geos::Geometry` | XYZ | - | [ToGeos] | [GeosWriter](geos::GeosWriter) |
//! | GPX | | XY | [GpxReader](gpx::GpxReader) | | |
//! | MVT | [mvt::tile::Feature] | XY | [mvt::tile::Layer] | [ToMvt] | [MvtWriter](mvt::MvtWriter) |
//! | SVG | - | XY | - | [ToSvg] | [SvgWriter](svg::SvgWriter) |
//! | WKB | [Wkb](wkb::Wkb), [Ewkb](wkb::Ewkb), [GpkgWkb](wkb::GpkgWkb), [SpatiaLiteWkb](wkb::SpatiaLiteWkb), [MySQL](wkb::MySQLWkb) | XYZM | - | [ToWkb] | [WkbWriter](wkb::WkbWriter) |
//! | WKT | [wkt::WktStr], [wkt::WktString], [wkt::EwktStr], [wkt::EwktString] | XYZM | [wkt::WktReader], [wkt::WktStr], [wkt::WktString], [wkt::EwktStr], [wkt::EwktString] | [ToWkt] | [WktWriter](wkt::WktWriter) |

#![warn(clippy::uninlined_format_args)]
#![allow(
Expand Down
18 changes: 18 additions & 0 deletions geozero/src/wkb/mod.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,24 @@ pub(crate) mod conversion {
) -> Result<Vec<u8>> {
self.to_wkb_dialect(WkbDialect::Geopackage, dims, srid, envelope)
}
/// Convert to Spatialite WKB.
fn to_spatialite_wkb(
&self,
dims: CoordDimensions,
srid: Option<i32>,
envelope: Vec<f64>,
) -> Result<Vec<u8>> {
self.to_wkb_dialect(WkbDialect::SpatiaLite, dims, srid, envelope)
}
/// Convert to MySQL WKB.
fn to_mysql_wkb(&self, srid: Option<i32>) -> Result<Vec<u8>> {
self.to_wkb_dialect(
WkbDialect::MySQL,
CoordDimensions::default(),
srid,
Vec::new(),
)
}
}

impl<T: GeozeroGeometry> ToWkb for T {
Expand Down
13 changes: 12 additions & 1 deletion geozero/src/wkb/wkb_common.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,13 @@ pub trait FromWkb {
}

/// WKB dialect.
#[derive(PartialEq, Debug)]
#[derive(PartialEq, Debug, Clone, Copy)]
pub enum WkbDialect {
Wkb,
Ewkb,
Geopackage,
MySQL,
SpatiaLite,
}

/// WKB Types according to OGC 06-103r4 (<https://www.ogc.org/standards/sfa>)
Expand Down Expand Up @@ -198,3 +200,12 @@ pub(crate) enum WKBByteOrder {
Xdr = 0, // Big Endian
Ndr = 1, // Little Endian
}

impl From<scroll::Endian> for WKBByteOrder {
fn from(endian: scroll::Endian) -> Self {
match endian {
scroll::BE => WKBByteOrder::Xdr,
scroll::LE => WKBByteOrder::Ndr,
}
}
}
Loading

0 comments on commit a92ef80

Please sign in to comment.