🚀 FEAT: Add true geometry for new links

[i-am-sijia]

As a user, when adding new roads, l would like to give them true geometries, instead of having them as stick links.

The current user workflow to add new roads

Met Council:

1. Open the CUBE .net file, add new stick links and nodes in CUBE, save out the .log file
2. Run Lasso to create Add New Roadway project cards from the .log file. A typical new roadway project card (e.g., an Met Council example in application) looks like:

project: example new road
tags: ''
dependencies: ''
changes:
- category: Add New Roadway
  links:
  - A: 111
    B: 222
    model_link_id: 333
    name: fake new road
    roadway: tertiary
    drive_access: 1
    walk_access: 1
    bike_access: 1
    rail_only: 0
    bus_only: 0
    lanes: 1
  nodes:
  - model_node_id: 111
    drive_access: 1
    walk_access: 1
    bike_access: 1
    rail_only: 0
    X: -93.04409
    Y: 44.948720
  - model_node_id: 222
    drive_access: 1
    walk_access: 1
    bike_access: 1
    rail_only: 0
    X: -93.05000
    Y: 44.950000

3. Sometimes users choose to skip the coding in CUBE, instead they create the new roadway project card manually following the above format.
4. Run Network Wrangler to apply the new roadway project card. Network Wrangler creates the new links and nodes with attributes specified in the project card, and stick link geometries between nodes.

Ideas for solution

Option	Pros	Cons
Option A: project card	Uses the existing Add New Roadway project card schema	More manual because it requires the user to edit the Add New Roadway project card. Also requires additional Lasso and Network Wrangler code changes.
Option B: wrangler card with input true shapes for the new links	More flexible, good for batch editing. Does not require additional Lasso or Network Wrangler code. User can input geometry using shapefile, geojson, etc.	Not a schema, harder to standardize
Option C: wrangler card with two input shapefiles	Same as Option B, except that user will input two network shapefiles, one from the CUBE export, one from the CUBE export after editing true shapes in GIS. Both shapefiles are of the same network with links more than just the true shape edited ones, to ensure connectivity	Not a schema, harder to standardize

Option A: Project Card

In the Add New Roadway project card, add geometry as a property under links, as a list of tuples (x,y coordinates).

project: example new road
tags: ''
dependencies: ''
changes:
- category: Add New Roadway
  links:
  - A: 111
    B: 222
    model_link_id: 333
    name: fake new road
    roadway: tertiary
    drive_access: 1
    walk_access: 1
    bike_access: 1
    rail_only: 0
    bus_only: 0
    lanes: 1
    geometry:
      - (-93.04409, 44.948720)
      - (-93.04609, 44.948820)
      - (-93.04809, 44.949020)
      - (-93.05000, 44.950000)
  nodes:
  - model_node_id: 111
    drive_access: 1
    walk_access: 1
    bike_access: 1
    rail_only: 0
    X: -93.04409
    Y: 44.948720
  - model_node_id: 222
    drive_access: 1
    walk_access: 1
    bike_access: 1
    rail_only: 0
    X: -93.05000
    Y: 44.950000

Option B: Wrangler Card

Use a roadway change wrangler card to overwrite stick geometries

---
project: example
tags:
dependencies:
category: Calculated Roadway
---

import geopandas as gpd
from NetworkWrangler import RoadwayNetwork
from NetworkWrangler.utils import create_unique_shape_id

# read input geometry file supplied by user, can be shapefiles, geojsons, the geometry to replace the base network with
new_geometry_gdf = gpd.read_file('xxxxx.geojson')

# convert the new_geometry to the same CRS as the base network
new_geometry_gdf = new_geometry_gdf.to_crs(self.shapes_df.crs)

# create unique shape hash for new geometry
new_geometry_gdf[RoadwayNetwork.UNIQUE_SHAPE_KEY] = new_geometry_gdf[
    "geometry"
].apply(lambda x: create_unique_shape_id(x))

# drop new geometry if 'model_link_id' is not in the base network
new_geometry_gdf = new_geometry_gdf[new_geometry_gdf['model_link_id'].isin(self.links_df['model_link_id'])]

# overwrite the base network with the new geometry, if it exists
self.links_df = self.links_df.set_index('model_link_id')
new_geometry_gdf = new_geometry_gdf.set_index('model_link_id')
# replace the geometry, and unique shape hash
self.links_df['geometry'] = new_geometry_gdf['geometry']
self.links_df[RoadwayNetwork.UNIQUE_SHAPE_KEY] = new_geometry_gdf[RoadwayNetwork.UNIQUE_SHAPE_KEY]

# reset the index
self.links_df = self.links_df.reset_index()
new_geometry_gdf = new_geometry_gdf.reset_index()

# add the new shape records to shapes
self.shapes_df = self.shapes_df.append(
    new_geometry_gdf[
        [RoadwayNetwork.UNIQUE_SHAPE_KEY, 'geometry']
    ]
)

Other considerations

• The first and last (X,Y) coordinates in the link geometry should match the end nodes
• ...

[i-am-sijia]

Initial thoughts on true geometry. We can discuss at tomorrow's meeting. @DavidOry @e-lo @RachelWikenMC @yueshuaing

[RachelWikenMC]

option c - would it be possible to make that work without requiring the full network for both shapefiles? The full network with all the walk links is VERY large and is difficult to work with. I always clip it down to a county or just drive links, etc, before doing any editing. Requiring the edits to be on entire network in a shapefile format will be very slow / impossible for editing

[RachelWikenMC]

Our conversation about this at the last meeting was choppy because of my tech issues. I'm fine with the wrangler file method, even though it requires a few steps of editing, I don't think we will use this feature for every edit. But it will be key for large complicated interchange projects.
In option B would I be running one wrangler file for each link id that needs new geometry? Or could I feed in a .json file with a number of new links?
Could the new geometry be in a different format - like .shp or does it have to be .json? I'm not using .json for my work currently so would have to figure out that conversion.

[i-am-sijia]

option c - would it be possible to make that work without requiring the full network for both shapefiles? The full network with all the walk links is VERY large and is difficult to work with. I always clip it down to a county or just drive links, etc, before doing any editing. Requiring the edits to be on entire network in a shapefile format will be very slow / impossible for editing

Sure. It can work with subset of network, as long as the two input shapefiles have the same links. The only thing it does is to compare the geometries from the two input shapefiles to find the true shapes user edited. a subset network is only better since it's faster to read in and compare.

[i-am-sijia]

In option B would I be running one wrangler file for each link id that needs new geometry? Or could I feed in a .json file with a number of new links?

You can feed in a number of links in one file. I think preferably a .shp, .geojson, instead of .json.

Option C is better than Option B because Option B's .shp only has the new shapes which is more prone to have them disconnected from the rest of the network. In Option C user first exports the network around the true shape into .shp, and then makes edits on the exported .shp for the true shapes, which helps maintain the connectivity.

[i-am-sijia]

Option C template:

---
project: example true geometry
tags: []
dependencies: {}
self_obj_type: RoadwayNetwork
note: ''
category: Calculated Roadway
---

# this is an example wrangler card to implement Option C dicussed in this issue:
# https://github.com/network-wrangler/projectcard/issues/15
# users will supply two shapefiles, both includes model_link_id and geometry
# the only difference they are trying to capture is the geometry for some links

import geopandas as gpd
import pandas as pd
import copy
from network_wrangler.utils.ids import create_str_int_combo_ids

## UPDATE HERE: in case the link id in the input .shp is not called model_link_id
UNIQUE_INPUT_LINK_ID_KEY = "link_id"
UNIQUE_SHAPE_KEY = "shape_id"

## copy the link and shape attrs in case any dataframe changes remove these attrs
links_attrs = copy.deepcopy(self.links_df.attrs)
shapes_attrs = copy.deepcopy(self.shapes_df.attrs)

########
# INPUTS
########

# read input geometry file supplied by user, can be shapefiles, geojsons

## UPDATE HERE: shapefile 1
## this is the link .shp before user changes any geometry
links_gdf = gpd.read_file("BaseLinks_head2.shp")

## UPDATE HERE: shapefile 2
## this is the link .shp after user changes any geometry
links_geometry_gdf = gpd.read_file("BaseLinks_head2_geometry_changes.shp")

## set index
links_gdf = links_gdf.set_index(UNIQUE_INPUT_LINK_ID_KEY).sort_index()
links_geometry_gdf = links_geometry_gdf.set_index(UNIQUE_INPUT_LINK_ID_KEY).sort_index()

## checking consistency
## check the two input shapefiles are of the same length
assert len(links_gdf)==len(links_geometry_gdf), "the two input files have different length"
## check the two input shapefiles have the same index, ignore sequence
assert links_gdf.index.equals(links_geometry_gdf.index), "the two input files have different links"

#########
# PROCESS
#########

# convert the two input files to the same CRS as the base network
links_gdf = links_gdf.to_crs("epsg:4269")
links_geometry_gdf = links_geometry_gdf.to_crs("epsg:4269")

# find links with different geometry
new_geometry_gdf = links_geometry_gdf[~links_geometry_gdf.geom_equals(links_gdf)]

# create unique shape hash for new geometry
new_geometry_gdf[UNIQUE_SHAPE_KEY] = create_str_int_combo_ids(
    len(new_geometry_gdf), self.shapes_df[UNIQUE_SHAPE_KEY]
)

# reset the index and rename it as "model_link_id"
new_geometry_gdf = new_geometry_gdf.reset_index().rename(columns={UNIQUE_INPUT_LINK_ID_KEY: "model_link_id"})

# overwrite the base network with the new geometry, if it exists
self.links_df = self.links_df.set_index('model_link_id')
new_geometry_gdf = new_geometry_gdf.set_index('model_link_id')
# replace the geometry, and unique shape hash
self.links_df['geometry'].update(new_geometry_gdf['geometry'])
self.links_df[UNIQUE_SHAPE_KEY].update(new_geometry_gdf[UNIQUE_SHAPE_KEY])

# reset and drop the index
self.links_df = self.links_df.reset_index()
assert "model_link_id" in self.links_df.columns
new_geometry_gdf = new_geometry_gdf.reset_index()

# add the new shape records to shapes
new_geometry_gdf = new_geometry_gdf.to_crs(self.shapes_df.crs)
self.shapes_df = pd.concat(
    [self.shapes_df, new_geometry_gdf[[UNIQUE_SHAPE_KEY, "geometry"]]]
)
# drop duplicate unique shape id, keep last 
self.shapes_df = self.shapes_df.drop_duplicates(subset=[UNIQUE_SHAPE_KEY], keep='last')

# make sure the attrs stay the same
self.links_df.attrs = links_attrs
self.shapes_df.attrs = shapes_attrs