Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing an option for the user to decide on simplifying GADM shapes #1138

Merged
merged 14 commits into from
Dec 20, 2024
Merged

Introducing an option for the user to decide on simplifying GADM shapes #1138

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
e9ac465
docs(contributor): contrib-readme-action has updated readme
github-actions[bot] Oct 8, 2024
4993072
feature: Introducing configurable option to simplify GADM shapes
SermishaNarayana Oct 10, 2024
7e93084
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 10, 2024
e85f01d
docs(contributor): contrib-readme-action has updated readme
github-actions[bot] Oct 16, 2024
76ff842
Adding 'simplify_gadm' description to configtables
SermishaNarayana Oct 17, 2024
a1a3042
doc: Release note update
SermishaNarayana Oct 17, 2024
60b9453
Merge branch 'pypsa-meets-earth:main' into main
SermishaNarayana Oct 17, 2024
b9b766f
docs(contributor): contrib-readme-action has updated readme
github-actions[bot] Oct 17, 2024
5c5fa5d
Merge branch 'main' of https://github.com/pypsa-meets-earth/pypsa-ear…
SermishaNarayana Oct 17, 2024
ce6bb81
Merge branch 'main' of github.com:SermishaNarayana/pypsa-earth into s…
SermishaNarayana Oct 17, 2024
f509683
Merge branch 'simplify_gadm' of github.com:SermishaNarayana/pypsa-ear…
SermishaNarayana Oct 17, 2024
a52deb0
Merge branch 'main' into simplify_gadm
SermishaNarayana Dec 20, 2024
953d428
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 20, 2024
accedd6
Set simplify_gadm to true by default
davide-f Dec 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 77 additions & 84 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,17 +191,17 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<table>
<tr>
<td align="center">
<a href="https://github.com/FabianHofmann">
<img src="https://avatars.githubusercontent.com/u/19226431?v=4" width="100;" alt="FabianHofmann"/>
<a href="https://github.com/SermishaNarayana">
<img src="https://avatars.githubusercontent.com/u/156903227?v=4" width="100;" alt="SermishaNarayana"/>
<br />
<sub><b>Fabian Hofmann</b></sub>
<sub><b>Null</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/fneum">
<img src="https://avatars.githubusercontent.com/u/29101152?v=4" width="100;" alt="fneum"/>
<a href="https://github.com/davide-f">
<img src="https://avatars.githubusercontent.com/u/67809479?v=4" width="100;" alt="davide-f"/>
<br />
<sub><b>Fabian Neumann</b></sub>
<sub><b>Davide-f</b></sub>
</a>
</td>
<td align="center">
Expand All @@ -212,60 +212,39 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
</a>
</td>
<td align="center">
<a href="https://github.com/euronion">
<img src="https://avatars.githubusercontent.com/u/42553970?v=4" width="100;" alt="euronion"/>
<a href="https://github.com/pz-max">
<img src="https://avatars.githubusercontent.com/u/61968949?v=4" width="100;" alt="pz-max"/>
<br />
<sub><b>Euronion</b></sub>
<sub><b>Max Parzen</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/Justus-coded">
<img src="https://avatars.githubusercontent.com/u/44394641?v=4" width="100;" alt="Justus-coded"/>
<a href="https://github.com/DeniseGiub">
<img src="https://avatars.githubusercontent.com/u/113139589?v=4" width="100;" alt="DeniseGiub"/>
<br />
<sub><b>Justus Ilemobayo</b></sub>
<sub><b>DeniseGiub</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/mnm-matin">
<img src="https://avatars.githubusercontent.com/u/45293386?v=4" width="100;" alt="mnm-matin"/>
<a href="https://github.com/yerbol-akhmetov">
<img src="https://avatars.githubusercontent.com/u/113768325?v=4" width="100;" alt="yerbol-akhmetov"/>
<br />
<sub><b>Mnm-matin</b></sub>
<sub><b>Yerbol Akhmetov</b></sub>
</a>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/martacki">
<img src="https://avatars.githubusercontent.com/u/53824825?v=4" width="100;" alt="martacki"/>
<br />
<sub><b>Martha Frysztacki</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/LukasFrankenQ">
<img src="https://avatars.githubusercontent.com/u/55196140?v=4" width="100;" alt="LukasFrankenQ"/>
<br />
<sub><b>Lukas Franken</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/pz-max">
<img src="https://avatars.githubusercontent.com/u/61968949?v=4" width="100;" alt="pz-max"/>
<br />
<sub><b>Max Parzen</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/davide-f">
<img src="https://avatars.githubusercontent.com/u/67809479?v=4" width="100;" alt="davide-f"/>
<a href="https://github.com/GbotemiB">
<img src="https://avatars.githubusercontent.com/u/48842684?v=4" width="100;" alt="GbotemiB"/>
<br />
<sub><b>Davide-f</b></sub>
<sub><b>Emmanuel Bolarinwa</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/koen-vg">
<img src="https://avatars.githubusercontent.com/u/74298901?v=4" width="100;" alt="koen-vg"/>
<a href="https://github.com/mnm-matin">
<img src="https://avatars.githubusercontent.com/u/45293386?v=4" width="100;" alt="mnm-matin"/>
<br />
<sub><b>Koen Van Greevenbroek</b></sub>
<sub><b>Mnm-matin</b></sub>
</a>
</td>
<td align="center">
Expand All @@ -274,57 +253,35 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<br />
<sub><b>Hazem</b></sub>
</a>
</td></tr>
<tr>
</td>
<td align="center">
<a href="https://github.com/energyLS">
<img src="https://avatars.githubusercontent.com/u/89515385?v=4" width="100;" alt="energyLS"/>
<br />
<sub><b>EnergyLS</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/AnasAlgarei">
<img src="https://avatars.githubusercontent.com/u/101210563?v=4" width="100;" alt="AnasAlgarei"/>
<br />
<sub><b>AnasAlgarei</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/yerbol-akhmetov">
<img src="https://avatars.githubusercontent.com/u/113768325?v=4" width="100;" alt="yerbol-akhmetov"/>
<br />
<sub><b>Yerbol Akhmetov</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/DeniseGiub">
<img src="https://avatars.githubusercontent.com/u/113139589?v=4" width="100;" alt="DeniseGiub"/>
<br />
<sub><b>DeniseGiub</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/GbotemiB">
<img src="https://avatars.githubusercontent.com/u/48842684?v=4" width="100;" alt="GbotemiB"/>
<br />
<sub><b>Emmanuel Bolarinwa</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/Tomkourou">
<img src="https://avatars.githubusercontent.com/u/5240283?v=4" width="100;" alt="Tomkourou"/>
<br />
<sub><b>Thomas Kouroughli</b></sub>
</a>
</td></tr>
<tr>
</td>
<td align="center">
<a href="https://github.com/GridGrapher">
<img src="https://avatars.githubusercontent.com/u/127969728?v=4" width="100;" alt="GridGrapher"/>
<br />
<sub><b>GridGrapher</b></sub>
</a>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/martacki">
<img src="https://avatars.githubusercontent.com/u/53824825?v=4" width="100;" alt="martacki"/>
<br />
<sub><b>Martha Frysztacki</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/finozzifa">
Expand Down Expand Up @@ -369,6 +326,27 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<sub><b>Cschau</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/euronion">
<img src="https://avatars.githubusercontent.com/u/42553970?v=4" width="100;" alt="euronion"/>
<br />
<sub><b>Euronion</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/AnasAlgarei">
<img src="https://avatars.githubusercontent.com/u/101210563?v=4" width="100;" alt="AnasAlgarei"/>
<br />
<sub><b>AnasAlgarei</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/LukasFrankenQ">
<img src="https://avatars.githubusercontent.com/u/55196140?v=4" width="100;" alt="LukasFrankenQ"/>
<br />
<sub><b>Lukas Franken</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/Tooblippe">
<img src="https://avatars.githubusercontent.com/u/805313?v=4" width="100;" alt="Tooblippe"/>
Expand All @@ -382,14 +360,22 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<br />
<sub><b>Anton Achhammer</b></sub>
</a>
</td>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/asolavi">
<img src="https://avatars.githubusercontent.com/u/131155817?v=4" width="100;" alt="asolavi"/>
<br />
<sub><b>Null</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/koen-vg">
<img src="https://avatars.githubusercontent.com/u/74298901?v=4" width="100;" alt="koen-vg"/>
<br />
<sub><b>Koen Van Greevenbroek</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/carlosfv92">
<img src="https://avatars.githubusercontent.com/u/103258059?v=4" width="100;" alt="carlosfv92"/>
Expand All @@ -403,8 +389,7 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<br />
<sub><b>Stephen J Lee</b></sub>
</a>
</td></tr>
<tr>
</td>
<td align="center">
<a href="https://github.com/rsparks3">
<img src="https://avatars.githubusercontent.com/u/30065966?v=4" width="100;" alt="rsparks3"/>
Expand All @@ -418,7 +403,8 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<br />
<sub><b>Null</b></sub>
</a>
</td>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/juli-a-ko">
<img src="https://avatars.githubusercontent.com/u/126512394?v=4" width="100;" alt="juli-a-ko"/>
Expand Down Expand Up @@ -446,8 +432,7 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<br />
<sub><b>Pietro Monticone</b></sub>
</a>
</td></tr>
<tr>
</td>
<td align="center">
<a href="https://github.com/Netotse">
<img src="https://avatars.githubusercontent.com/u/89367243?v=4" width="100;" alt="Netotse"/>
Expand All @@ -461,7 +446,8 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<br />
<sub><b>Katherine M. Antonio</b></sub>
</a>
</td>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/jessLryan">
<img src="https://avatars.githubusercontent.com/u/122939887?v=4" width="100;" alt="jessLryan"/>
Expand All @@ -483,21 +469,28 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<sub><b>HanaElattar</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/FabianHofmann">
<img src="https://avatars.githubusercontent.com/u/19226431?v=4" width="100;" alt="FabianHofmann"/>
<br />
<sub><b>Fabian Hofmann</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/EmreYorat">
<img src="https://avatars.githubusercontent.com/u/93644024?v=4" width="100;" alt="EmreYorat"/>
<br />
<sub><b>EmreYorat</b></sub>
</a>
</td></tr>
<tr>
</td>
<td align="center">
<a href="https://github.com/AndreCNF">
<img src="https://avatars.githubusercontent.com/u/19359510?v=4" width="100;" alt="AndreCNF"/>
<br />
<sub><b>André Cristóvão Neves Ferreira</b></sub>
</a>
</td>
</td></tr>
<tr>
<td align="center">
<a href="https://github.com/AlexanderMeisinger">
<img src="https://avatars.githubusercontent.com/u/91368938?v=4" width="100;" alt="AlexanderMeisinger"/>
Expand Down
1 change: 1 addition & 0 deletions config.default.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ cluster_options:

build_shape_options:
gadm_layer_id: 1 # GADM level area used for the gadm_shapes. Codes are country-dependent but roughly: 0: country, 1: region/county-like, 2: municipality-like
simplify_gadm: false # When true, shape polygons are simplified else no
davide-f marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default, let's enable the simplification. This on average speeds up computation and is desirable

update_file: false # When true, all the input files are downloaded again and replace the existing files
out_logging: true # When true, logging is printed to console
year: 2020 # reference year used to derive shapes, info on population and info on GDP
Expand Down
34 changes: 22 additions & 12 deletions scripts/build_shapes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,7 @@ def eez(
distance=0.01,
minarea=0.01,
tolerance=0.01,
simplify_gadm=True,
):
"""
Creates offshore shapes by buffer smooth countryshape (=offset country
Expand All @@ -386,22 +387,26 @@ def eez(
}
).set_index("name")

ret_df = ret_df.geometry.map(
lambda x: _simplify_polys(x, minarea=minarea, tolerance=tolerance)
)
if simplify_gadm:
ret_df = ret_df.geometry.map(
lambda x: _simplify_polys(x, minarea=minarea, tolerance=tolerance)
)

ret_df = ret_df.apply(lambda x: make_valid(x))
ret_df = ret_df.apply(lambda x: make_valid(x))

country_shapes_with_buffer = country_shapes.buffer(distance)
ret_df_new = ret_df.difference(country_shapes_with_buffer)

# repeat to simplify after the buffer correction
ret_df_new = ret_df_new.map(
lambda x: (
x if x is None else _simplify_polys(x, minarea=minarea, tolerance=tolerance)
if simplify_gadm:
# repeat to simplify after the buffer correction
ret_df_new = ret_df_new.map(
lambda x: (
x
if x is None
else _simplify_polys(x, minarea=minarea, tolerance=tolerance)
)
)
)
ret_df_new = ret_df_new.apply(lambda x: x if x is None else make_valid(x))
ret_df_new = ret_df_new.apply(lambda x: x if x is None else make_valid(x))

# Drops empty geometry
ret_df = ret_df_new.dropna()
Expand Down Expand Up @@ -1256,6 +1261,7 @@ def gadm(
out_logging=False,
year=2020,
nprocesses=None,
simplify_gadm=True,
):
if out_logging:
logger.info("Stage 3 of 5: Creation GADM GeoDataFrame")
Expand Down Expand Up @@ -1305,7 +1311,9 @@ def gadm(
lambda x: x if x.find(".") == 0 else "." + x
)
df_gadm.set_index("GADM_ID", inplace=True)
df_gadm["geometry"] = df_gadm["geometry"].map(_simplify_polys)

if simplify_gadm:
df_gadm["geometry"] = df_gadm["geometry"].map(_simplify_polys)
df_gadm.geometry = df_gadm.geometry.apply(
lambda r: make_valid(r) if not r.is_valid else r
)
Expand Down Expand Up @@ -1338,6 +1346,7 @@ def gadm(
contended_flag = snakemake.params.build_shape_options["contended_flag"]
worldpop_method = snakemake.params.build_shape_options["worldpop_method"]
gdp_method = snakemake.params.build_shape_options["gdp_method"]
simplify_gadm = snakemake.params.build_shape_options["simplify_gadm"]

country_shapes = countries(
countries_list,
Expand All @@ -1349,7 +1358,7 @@ def gadm(
country_shapes.to_file(snakemake.output.country_shapes)

offshore_shapes = eez(
countries_list, geo_crs, country_shapes, EEZ_gpkg, out_logging
countries_list, geo_crs, country_shapes, EEZ_gpkg, out_logging, simplify_gadm
)

offshore_shapes.reset_index().to_file(snakemake.output.offshore_shapes)
Expand All @@ -1371,5 +1380,6 @@ def gadm(
out_logging,
year,
nprocesses=nprocesses,
simplify_gadm=simplify_gadm,
)
save_to_geojson(gadm_shapes, out.gadm_shapes)
Loading