Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Round 2 enhancements for Felt #47

Merged
merged 86 commits into from
Jul 20, 2023
Merged
Show file tree
Hide file tree
Changes from 81 commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
2ecc9fc
bug fix for town, village as city level features (not neighbourhood)
nvkelso Jun 15, 2023
4646d8f
init Makefile targets
nvkelso Jun 15, 2023
bdcd8c1
add pmap:kind coallesce
nvkelso Jun 15, 2023
0e45322
converge towards shield_text_length; pass thru other network values; …
nvkelso Jun 15, 2023
99bd2df
refactor pmap:kind calculation and add pmap:kind_detail
nvkelso Jun 15, 2023
3a8ad37
refactor places, add pmap:min_zoom, add sort func, add label grid
nvkelso Jun 15, 2023
bda1226
rework low zoom NE water
nvkelso Jun 15, 2023
cce8016
stub out water label points (natural earth only)
nvkelso Jun 16, 2023
39e9092
fix test tile coord; tmp pmtiles bin location
nvkelso Jun 21, 2023
a33723f
add kind = national_park logic
nvkelso Jun 21, 2023
330ad89
add derived water polygon labels in physical points layer
nvkelso Jun 21, 2023
dedcc61
fix java errors, comment out setSortKey for later debug
nvkelso Jun 21, 2023
d452bbe
reorder sections for a-z legibility, add optional pmap:kind_detail pr…
nvkelso Jun 21, 2023
0faa99d
add washington state target
nvkelso Jun 22, 2023
d6baab1
show county lines earlier, export Tilezen kinds (and region not state)
nvkelso Jun 22, 2023
89cc47c
block {building|building:part}=no; push building:part to later zooms;…
nvkelso Jun 22, 2023
f19aa7d
do NOT export names; formatting
nvkelso Jun 22, 2023
e501a32
do NOT export names, or areas
nvkelso Jun 22, 2023
1a34443
export label points for water polygons; & etc
nvkelso Jun 22, 2023
73249f6
use Tilezen kind values (locality not city)
nvkelso Jun 22, 2023
44f3cc2
export POIs for ways, rels too (area graded per zoom); some kind of b…
nvkelso Jun 22, 2023
a93e7b1
fix service minor roads; fix sidewalks
nvkelso Jun 22, 2023
dde2a3f
show piers later; drop short featues
nvkelso Jun 22, 2023
1dce93b
fix NE layer parsing; do NOT export names (see physical points layer …
nvkelso Jun 22, 2023
492e03c
add comment
nvkelso Jun 23, 2023
05c7890
quantize heights by zoom; push building_part to later zooms
nvkelso Jun 23, 2023
492919e
always export pmap:kind
nvkelso Jun 23, 2023
29839e6
add national_park, protected_area, and nature_reserve (moved from nat…
nvkelso Jun 23, 2023
1b91f74
remove national_park, protected_area, nature_reserve (moved to landus…
nvkelso Jun 23, 2023
153a6bd
show rivers earlier; always set kind value; intermittent to boolean
nvkelso Jun 23, 2023
fe8aa5a
switch to min_label for NE instead of min_zoom; update zoom area grad…
nvkelso Jun 23, 2023
dc6b9c1
prefix population_rank with pmap:
nvkelso Jun 23, 2023
bffdf4d
bug fixes for kind calculations; better way area calculation; show la…
nvkelso Jun 23, 2023
ba46ffe
show motorway and primary earlier; show service roads later; show pat…
nvkelso Jun 23, 2023
55c1dbf
show airport runway, taxiway earlier; show some types of rail later; …
nvkelso Jun 23, 2023
c73c1df
update kind calculation; boolean exports
nvkelso Jun 23, 2023
30ab35c
prefix population_rank with pmap:
nvkelso Jun 23, 2023
28a99c8
remove debug attr
nvkelso Jun 23, 2023
99a4c73
add phony list; remove complicated targets
nvkelso Jun 23, 2023
b56aafa
add Tilezen schema comments
nvkelso Jun 24, 2023
62cb079
add Tilezen schema comments
nvkelso Jun 24, 2023
79be151
add beach, forest, military, naval_base, airfield, zoo kinds; bug fix…
nvkelso Jun 24, 2023
45c11fc
move kind calc above; Tilezen schema notes; min pixel size to 2 from 3
nvkelso Jun 24, 2023
59d4995
add Tilezen schema comments
nvkelso Jun 24, 2023
f1b9478
some tiny lake labels earlier; add Tilezen schema comments
nvkelso Jun 24, 2023
78a6c78
refactor property gathering logic and move setAttr later; add Tilezen…
nvkelso Jun 24, 2023
6a9972c
add attraction kinds, beach kind, hide early node university kind, ad…
nvkelso Jun 24, 2023
553e5eb
refactor property gathering and attr setting; Tilezen schema notes
nvkelso Jun 24, 2023
2c7ab10
Tilezen schema notes; hide names from early zooms for more merging
nvkelso Jun 24, 2023
4d6e41e
refactor property gathering and attr setting; Tilezen schema notes
nvkelso Jun 24, 2023
433d6c2
remove unused imports, fix silly Java error
nvkelso Jun 24, 2023
88fb01c
add to early zoom blocklist
nvkelso Jun 24, 2023
fa265a0
maven formatting
nvkelso Jun 24, 2023
b27b857
update style for schema changes
nvkelso Jun 28, 2023
34ca7b3
add SF target
nvkelso Jun 28, 2023
b5a5b99
ensure pier pass thru
nvkelso Jun 28, 2023
27291b4
add grass
nvkelso Jun 28, 2023
22b7635
add label grid, separate NE from OSM at zoom 7
nvkelso Jun 28, 2023
e18abdf
add label grid; bug fixes for kinds and zooms (per kind)
nvkelso Jun 28, 2023
e0a56da
add planet target
nvkelso Jun 28, 2023
84ddc13
significant whitespace
nvkelso Jun 28, 2023
eed7401
typo
nvkelso Jun 28, 2023
19c529c
planet not world
nvkelso Jun 28, 2023
9fb4d01
add new targets
nvkelso Jun 29, 2023
06fe05c
use low zoom boundaries from Natural Earth
nvkelso Jun 29, 2023
b46e565
add zoom logic for country, region
nvkelso Jun 29, 2023
e68bd09
guard against nulls
nvkelso Jun 30, 2023
5fbd2a2
guard against null names; spell out more country names
nvkelso Jun 30, 2023
4581fc1
guard against null names
nvkelso Jun 30, 2023
d94aafb
perf; rework NE kind and kind_detail
nvkelso Jun 30, 2023
41256f1
npe, again; cleanup imports and var names
nvkelso Jul 1, 2023
580d998
standardize on var names
nvkelso Jul 1, 2023
ce14bef
ensure odd numbered admin_level do not export
nvkelso Jul 1, 2023
7739f7f
add brunnel pmap:level, remove dedicated props
nvkelso Jul 1, 2023
d95ab78
add brunnel; push disused lines later
nvkelso Jul 1, 2023
02da19c
add quarter back
nvkelso Jul 1, 2023
8b69a55
standardize sf var name
nvkelso Jul 1, 2023
36d1527
push small parks later; university buildings later
nvkelso Jul 1, 2023
3d06817
brunnel; standardize sf var name; kind_detail
nvkelso Jul 1, 2023
7babaad
regrade park area > zooms
nvkelso Jul 1, 2023
3c1789d
add changelog, version, and semver statement
nvkelso Jul 10, 2023
90f584b
similar casing order change as #42 for merge conflicts
nvkelso Jul 12, 2023
49e81d4
reserve merge conflict over road casing order #42
nvkelso Jul 12, 2023
262e9ec
linting
nvkelso Jul 12, 2023
5b0ae4e
update 1.0.0 date
bdon Jul 14, 2023
b7221a2
Merge branch 'main' into nvkelso/felt-changes-part-2
bdon Jul 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions CHANGELOG.md

Large diffs are not rendered by default.

200 changes: 200 additions & 0 deletions SEMANTIC-VERSIONING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
# Semantic Versioning

When a new version of the Protomaps basemap schema is released, developers should be able to tell from the
version increment how much effort it will take them to integrate the new tiles with their map. We use semantic
versioning to communicate this.

### What is Semantic Versioning?

Semantic versioning (or [SemVer](http://semver.org/)) is a formalized way of making promises with an X.Y.Z version indicator.

#### Version components

- `MAJOR`.`MINOR`.`PATCH`, example: `1.0.0` (default)
- `MAJOR`.`MINOR`.`PATCH`-`POSTFIX`, example: `1.0.0-pre1` (optional)

#### Version parts:

1. **MAJOR** version **X** for incompatible API changes.
2. **MINOR** version **Y** when adding functionality in a backwards-compatible manner, and
3. **PATCH** version **Z** when fixing backwards-compatible bugs
4. **pre-release** version **-POSTFIX** when releasing developer previews

**NOTE:** The above applies _after_ software has reached as version `1.0.0`, no promises are made for earlier versions.

#### Developer level of effort:

- Major version X: **high** – significant integration challenges, read the changelog closely
- Minor version Y: **low** – some integration challenges, read the changelog
- Bug fixes Z: **none** – simply use the new tiles, skim or ignore the changelog
- Pre-release POSTFIX: **low to high** – some to significant integration challenges, read the changelog

## Versioning the Protomaps Basemap vector tile schema

Upon our version `1.0.0` release Protomaps Basemap makes the following promises based on the Tilezen schema:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we prefer starting at 1.0.0 for semver? In relation to the hosted API urls:

api.protomaps.com/tiles/v2/ is major version 2 (the live one right now)

so api.protomaps.com/tiles/v2/{z}/{x}/{y}.mvt is the latest tileset for v2. These are the main stable URLs in the system.

api.protomaps.com/tiles/v2/20230701/{z}/{x}/{y}.mvt is a timestamped tileset compatible with the major version. it makes no guarantees about the minor version, but the full version string can be queried through metadata. Ideally we don't keep more than a couple months to a year of history, so these URLs should be treated as transient.

How should we address beta releases? api.protomaps.com/tiles/v2/20230701-beta/{z}/{x}/{y}.mvt ? These should reflect the -beta postfix in the metadata version string. Or should we use a different endpoint entirely?

I'd like to avoid building endpoints like v2/2.3.1 as I think every minor version should supersede all previous minor versions of that major version. If you wanted to find a specific minor release of the official build, we can build an index mapping timestamped tilesets to minor version.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is largely a copy-paste job from Tilezen, so there may be some details (some of which you've noted) to iron out still. Thanks for being thorough here...

SemVer is pretty explicate that once you hit 1.0.0 you've made a contract with the end-user with respect to future major-minor-patch versions.

So even though I'm conceptualizing this PR as being for Protomaps version 3.0.0 schema, I think this doc should still talk about 1.0.0.

Your description of how Protomaps API works is related to the version of the schema. Your description of /v2/ and /v2/20230701/ make sense to me.

I don't have strong opinions about how you setup the API, though I think it's useful to think about it in SemVer terms:

  • Next major version: /v3/20230701-beta/ or /v3/beta-v3.0.0-pre-20230701/
  • Next minor version: /v2/20230701-beta/ or /v2/beta-v2.5.0-pre-20230701/
  • Next patch version: /v2/20230701-beta/ or /v2/beta-v2.5.1-pre-20230701/

Where 20230701-beta is an arbitrary build date (and allow for an {a,b,c} variant) with text indicator that it's not production worthy. So the question is where to put metadata about what schema (pre)release or SHA for the code was used, what OSM data went in, what Natural Earth version was used – and in future what Who's On First version (and even a landcover source)...

All the beta releases shouldn't have any user facing contract – the are effectively ephemeral. But both incremental minor released versions and patch released versions and beta versions need to be allowed for under the same "/v2/" path and depending on developer velocity sometimes they may all be in flight at the same time.

l'll note here other discussion elsewhere in this repo about NPM package is more about the style, which is somewhat dependent on the schema. Which creates an awkward situation of having 2 different "versions" in the same repo. Hmm.


##### Definition of terms

* **`common`** - These `layer`s, `property`s, and `kind`s are generally available across all features in a Tilezen response.
- Establishes basic selection of features and their arrangement into specific named layers.
- Core properties needed for display and labeling of features:
- Special bits that make vector tile content **interoperably Tilezen**, including `kind`, `kind_detail`, `landuse_kind`, `kind_tile_rank`, `min_zoom`, `max_zoom`, `is_landuse_aoi`, `sort_rank`, `boundary`, and `maritime_boundary`.
- Fundamental properties like `name` (including localized names), `id`, and `source` included on most every feature.
* **`common-optional`** - These are meant to be part of a common set, but may not be present because they aren't relevant or because we don't have the data (primarily feature `property`s, but could also be `layer`s).
- Used to refine feature selection.
- Lightly transformed **interoperable Tilezen** properties based on original data values. Examples include: `country_capital`, `region_capital`, `bicycle_network`, `is_bridge`, `is_link`, `is_tunnel`, `is_bicycle_related`, `is_bus_route`, `walking_network`, `area`, left & right names and localized `name:*` values on lines, and left & right `id` values on lines.
- Fundamental properties like `ref`, `colour`, `population`, `elevation`, `cuisine`, `operator`, `protect_class`, and `sport`.
* **`optional`** - These are the properties of a specific, less important `kind`, or generally present across `kind`s but only in exceptional cases.
- Often used to decorate features already selected for display.
- Additional properties like `capacity` and `covered`.

#### MAJOR version increments:

1. **Remove** `common layer`
1. **Change** `common layer` **name**
1. **Remove** `common-optional layer`
1. **Change** `common-optional layer` **name**
1. **Remove** `common` feature **property** completely from all zooms
1. **Change** `common` feature **property name**
1. **Remove** `common-optional` feature **property** at zoom 14 or more
1. **Change** `common-optional` feature **property name**
1. **Remove** `optional` feature **property** at zoom 14 or more
1. **Change** `kind` **value name**
1. **Remove** `kind` **value** completely from all zooms
1. **Move** `kind` from one layer to another
1. **Additional simplification** across `kind` **values** in zooms 14, 15, or 16 (or max zoom) by removing `common`, `common-optional`, and/or `optional` **properties** by merging or other method
1. **Simplification** within `kind` **values** at zooms 16 (or max zoom) by removing `common`, `common-optional`, and `optional` **properties** by merging or other method
1. **Change** of <= -3 (earlier) to default `min_zoom` or `max_zoom` **values** to determine when `kind` is included
1. **Change** of >= +2 (later) to default `min_zoom` or `max_zoom` **values** to determine when `kind` is included

#### MINOR version increments:

1. **Add** `common` layer
1. **Add** `common-optional` layer
1. **Add** `common` feature **property**
1. **Add** `common-optional` feature **property**
1. **Add** `optional` feature **property**
1. **Add** new `kind` value
1. **Additional simplification** across `kind` **values** at zooms 13 or less by removing `common`, `common-optional`, and `optional` **properties** by merging or other method
1. **Additional simplification** within `kind` **values** at zooms 13, 14, or 15 by removing `common`, `common-optional`, and `optional` **properties** by merging or other method
1. **Reassign** 50% or more of existing `kind` **value** into a new `kind` **value**, when kind has 10,000 or more features
1. **Change** of <= -2 (earlier) to `min_zoom` or `max_zoom` **values** to determine when `kind` is included
1. **Change** of >= +1 (later) to default `min_zoom` or `max_zoom` **values** to determine when `kind` is included
1. **Change** the maximum Tilezen zoom (currently zoom 16).

#### PATCH version increments

1. **Add** `optional` layer
1. **Change** `optional` layer **name**
1. **Remove** `optional` layer
1. **Additional simplification** within `kind` **values** at zooms 12 or less by removing `common`, `common-optional`, and `optional` **properties** by merging or other method
1. **Reassign** an existing `kind` **value** to an another existing `kind` value when they are equivalent (e.g. fixing the spelling of a value coming from an upstream data source)
1. **Reassign** less than 50% of existing `kind` **value** into a new `kind` value, when kind has 10,000 or more features
1. **Reassign** more than 50% of existing `kind` **value** into a new `kind` value, when kind has less than 10,000 features
1. **Change** of -1 to default `min_zoom` or `max_zoom` **values** to determine when `kind` is included
1. **Adjustments** to the overall map balance (proportion of features in one layer or another, proportion of `kind`s in a single layer) by adjusting boosting of `min_zoom` values over the `kind`'s default, limiting the number of individual `kind` features in a given tile coordinate, and other means.
1. **Add** unpublicized `kind` **value**
1. **Remove** unpublicized `kind` **value**
1. **Correct** a regression in the API (to the last good version)
1. **Correct** a newly added feature **property name**
1. **Correct** a newly added `kind` **value**

## Versioning Data

We do not version data features, but we do attempt to indicate the data source and the feature ID as assigned by that source so customers can investigate upstream changes.


### Frequency of data updates

Tilezen has 4 primary sources:

- **Natural Earth** (used for zooms 0-8 for most everything) updates infrequently (often annually)
- **OpenStreetMap** (used for zooms 9+ for most everything, sometimes earlier) updates frequently (at least daily)
- **OpenStreetMapData** (used for zooms 9+ in the earth and water layers only) updates infrequently (optimistically monthly)
- **Who’s On First** (used for zooms 12+ for places layer) updates frequently (at least daily)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the inclusion of WOF change the attribution requirements?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WOF is not yet incorporated here in v3, that's a v4 task (or v3.1 if really ambitious). So this line should be removed or rephrased now.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #79 via b3a953b.



### INDIVIDUAL FEATURES are not versioned
bdon marked this conversation as resolved.
Show resolved Hide resolved

#### Examples:

1. **Add** new feature
1. **Removal** of existing feature
1. **Change** feature **name**
1. **Change** feature **geometry**
1. **Change** feature **property** values (including property removal if removed from original source)
1. **Change** feature `kind` **value** (when upstream data source reclassifies them).
1. **Change** feature `min_zoom` &/or `max_zoom` **values** (when area or other signal changes upstream).

**NOTE:** It is possible to query the version of individual features by looking at a feature's `source` and `id` properties and performing a lookup via the source service, but that is beyond the scope of Tilezen. Because of simplification, `id` properties are not always available due to feature merging.

### LANGUAGES are not versioned

In addition to the `common` **name** locals call a place, the following `common` and `common-optional` languages are generally available:

#### Common languages:

1. `name:ar` **العربية (Arabic)** _common_
1. `name:bn` **বাংলা (Bengali)** – _common-optional_
1. `name:de` **Deutsch (German)** – _common-optional_
1. `name:en` **English** – _common_
1. `name:es` **español (Spanish)** – _common_
1. `name:fr` **français (French)** – _common_
1. `name:el` **ελληνικά (Greek)** – _common-optional_
1. `name:hi` **हिन्दी (Hindi)** – _common-optional_
1. `name:id` **Bahasa Indonesia (Indonesian)** – _common-optional_
1. `name:it` **italiano (Italian)** – _common-optional_
1. `name:ja` **日本語 (Japanese)** – _common-optional_
1. `name:ko` **한국어 (Korean)** – _common-optional_
1. `name:nl` **Nederlands (Dutch)** – _common-optional_
1. `name:pl` **Polski (Polish)** – _common-optional_
1. `name:pt` **Português (Portuguese)** – _common-optional_
1. `name:ru` **Русский (Russian)** – _common_
1. `name:sv` **Svenska (Swedish)** – _common-optional_
1. `name:tr` **Türkçe (Turkish)** – _common-optional_
1. `name:vi` **Tiếng Việt (Vietnamese)** – _common-optional_
1. `name:zh` **中文 (Chinese)**: primarily simplified but sometimes traditional – _common_ – _deprecated_
1. `name:zh-Hans` **中文 (Chinese)**: primarily simplified but sometimes traditional – _common_
1. `name:zh-Hant` **中文 (Chinese)**: primarily traditional but sometimes simplified – _common_

Arabic, Chinese, English, French, Russian and Spanish are used by the United Nations for meetings and official documents. The other languages listed are either proposed as official language of the United Nations (Bengali, Hindi, Portugese, and Turkish) or frequently used in OpenStreetMap, Who's On First, or Wikipedia.

Additional localized names are available as `common-optional` and `optional`, but their actual use is the data is not widespread.


### POLITICAL GEOGRAPHY is not versioned

#### Examples:

1. **Major** additions, deletions to country names, borders, disputed territories, and capitals are possible and may be advertised but do not bump the Tilezen API version.
1. **Minor** corrections to country names, borders, disputed territories, capitals, and other administrative geography are always possible and will not be tracked or advertised.

## Versioning the Tilezen Service

#### MAJOR version increments:

1. **Rollup** of MAJOR Tilezen API changes
1. **Change** the maximum Tilezen zoom at which tiles are generated by default (currently zoom 16)
1. **Remove** a file format
1. Backwards incompatible change to a file format
1. Tilezen software dependency has a breaking change to the tile response

#### MINOR version increments:

1. **Rollup** of MINOR Tilezen API changes
1. **Add** a file format

#### PATCH version increments:

1. **Rollup** of PATCH Tilezen API changes
1. **Infrequent update** of static data sources like Natural Earth or OpenStreetMapData
1. Backwards compatible change to a file format
1. Software dependency has a backwards compatible change or bug fix with no change to the tile response

## See also

- [VERSION](VERSION)
- http://semver.org
- https://github.com/tilezen/vector-datasource/edit/master/docs/SEMANTIC-VERSIONING.md
- https://github.com/nvkelso/natural-earth-vector/blob/master/README.md
- https://github.com/whosonfirst/whosonfirst-placetypes#roles
1 change: 1 addition & 0 deletions VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v3.0.0-pre1
Loading