Planet generation [3h15m]

[lokkelvin2]

Wanted to share my stats for generation of planet-latest.osm.pbf, using my OMT fork (which includes extra POI layers over the standard OMT schema)

Bare metal: Xeon-8168 (24c/48t), 192GB ram, 500GB SSD
Total time 3h15m, mbtile size 114GB.

I might experiment with --building-merge-z13=false to see how much time it shaves.

planet_generation.log

3:15:42 DEB [mbtiles] - Tile stats:
3:15:42 DEB [mbtiles] - z0 avg:69k max:69k
3:15:42 DEB [mbtiles] - z1 avg:163k max:184k
3:15:42 DEB [mbtiles] - z2 avg:467k max:844k
3:15:42 DEB [mbtiles] - z3 avg:152k max:817k
3:15:42 DEB [mbtiles] - z4 avg:56k max:668k
3:15:42 DEB [mbtiles] - z5 avg:28k max:619k
3:15:42 DEB [mbtiles] - z6 avg:14k max:374k
3:15:42 DEB [mbtiles] - z7 avg:11k max:408k
3:15:42 DEB [mbtiles] - z8 avg:6.3k max:341k
3:15:42 DEB [mbtiles] - z9 avg:5.8k max:461k
3:15:42 DEB [mbtiles] - z10 avg:2.6k max:319k
3:15:42 DEB [mbtiles] - z11 avg:1.2k max:187k
3:15:42 DEB [mbtiles] - z12 avg:720 max:238k
3:15:42 DEB [mbtiles] - z13 avg:392 max:266k
3:15:42 DEB [mbtiles] - z14 avg:482 max:3.6M
3:15:42 DEB [mbtiles] - all avg:500 max:0
3:15:42 DEB [mbtiles] -  # features: 3,827,695,863
3:15:42 DEB [mbtiles] -     # tiles: 267,485,076
3:15:42 INF [mbtiles] - Finished in 1h1m5s cpu:32h58m44s gc:3m7s avg:32.4
3:15:42 INF [mbtiles] -   read    1x(52% 31m37s sys:3m48s wait:25m50s done:22s)
3:15:42 INF [mbtiles] -   encode 48x(62% 37m58s sys:2m58s wait:4m28s done:7s)
3:15:42 INF [mbtiles] -   write   1x(32% 19m49s sys:6m47s wait:39m23s)
3:15:43 INF - Finished in 3h15m44s cpu:79h26m50s gc:30m26s avg:24.4
3:15:43 INF - FINISHED!
3:15:43 INF -
3:15:43 INF - ----------------------------------------
3:15:43 INF -   overall          3h15m44s cpu:79h26m50s gc:30m26s avg:24.4
3:15:43 INF -   lake_centerlines 0.5s cpu:3s avg:6.7
3:15:43 INF -   water_polygons   1m19s cpu:42m56s gc:1s avg:32.5
3:15:43 INF -     read     1x(59% 47s sys:27s wait:26s done:4s)
3:15:43 INF -     process 48x(64% 51s sys:26s wait:26s)
3:15:43 INF -     write    1x(47% 37s sys:6s wait:41s)
3:15:43 INF -   natural_earth    8s cpu:26s avg:3.2
3:15:43 INF -     read     1x(75% 6s done:1s)
3:15:43 INF -     process 48x(3% 0.2s wait:7s)
3:15:43 INF -     write    1x(0% 0s wait:7s)
3:15:43 INF -   osm_pass1        19m44s cpu:3h46m30s gc:4m18s avg:11.5
3:15:43 INF -     read     1x(5% 1m3s sys:34s wait:17m46s done:7s)
3:15:43 INF -     parse   46x(6% 1m10s sys:11s wait:17m55s done:2s)
3:15:43 INF -     process  1x(77% 15m14s sys:1m1s wait:15s)
3:15:43 INF -   osm_pass2        1h44m57s cpu:40h30m2s gc:22m24s avg:23.2
3:15:43 INF -     read     1x(2% 1m38s sys:43s wait:1h39m2s done:3m54s)
3:15:43 INF -     process 47x(40% 42m27s sys:14m39s block:21m23s wait:7m37s)
3:15:43 INF -     write    1x(34% 35m47s sys:6m32s wait:47m17s)
3:15:43 INF -   boundaries       13s cpu:17s avg:1.2
3:15:43 INF -   sort             7m30s cpu:1h27m9s gc:35s avg:11.6
3:15:43 INF -     worker 48x(15% 1m6s sys:7s wait:5m16s done:27s)
3:15:43 INF -   mbtiles          1h1m5s cpu:32h58m44s gc:3m7s avg:32.4
3:15:43 INF -     read    1x(52% 31m37s sys:3m48s wait:25m50s done:22s)
3:15:43 INF -     encode 48x(62% 37m58s sys:2m58s wait:4m28s done:7s)
3:15:43 INF -     write   1x(32% 19m49s sys:6m47s wait:39m23s)
3:15:43 INF - ----------------------------------------
3:15:43 INF -   features        245GB
3:15:43 INF -   mbtiles 114GB

[msbarry]

Nice! Thanks for posting. What command did you run to kick off planetiler? I'm curious what the memory settings were, it looked like there's more GC than I'd expect - 30 minutes vs 1 minute from the most recent run here where the jvm got 100g of ram.

Looks like all the buildings made z14 tiles pretty big (up to 3.6mb). Might be helpful to eventually support generating out to z15/16 to include more very fine details when zoomed way in.