Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datashader #2883

Merged
merged 48 commits into from
Sep 20, 2023
Merged

Datashader #2883

merged 48 commits into from
Sep 20, 2023

Conversation

jkrumbiegel
Copy link
Member

@jkrumbiegel jkrumbiegel commented Apr 21, 2023

Copied the prototype from https://github.com/cjdoris/ShadeYourData.jl over and modified it into a normal recipe. It's pretty slow when trackpad zooming (too many updates at once), but ok with the rectangle zoom.

using GLMakie
using DelimitedFiles

GLMakie.activate!(float = true)

airports = Point2f.(eachrow(readdlm(assetpath("airportlocations.csv"))))

fig, ax, ds = datashader(airports,
    colormap = [:black; Makie.to_colormap(:batlow)],
    binfactor = 2,
    post = mat -> log10.(mat .+ 1f0),
    axis = (; aspect = DataAspect())
)

datashader

@MakieBot
Copy link
Collaborator

MakieBot commented Apr 21, 2023

Compile Times benchmark

Note, that these numbers may fluctuate on the CI servers, so take them with a grain of salt. All benchmark results are based on the mean time and negative percent mean faster than the base branch. Note, that GLMakie + WGLMakie run on an emulated GPU, so the runtime benchmark is much slower. Results are from running:

using_time = @ctime using Backend
# Compile time
create_time = @ctime fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @ctime Makie.colorbuffer(display(fig))
# Runtime
create_time = @benchmark fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @benchmark Makie.colorbuffer(display(fig))
using create display create display
GLMakie 11.04s (10.90, 11.26) 0.11+- 1.12s (1.10, 1.14) 0.01+- 801.28ms (776.45, 853.50) 24.42+- 9.63ms (9.51, 9.71) 0.08+- 139.93ms (137.84, 145.89) 2.86+-
master 10.99s (10.90, 11.12) 0.08+- 1.11s (1.10, 1.14) 0.01+- 786.18ms (778.87, 790.43) 3.61+- 9.73ms (9.65, 9.80) 0.05+- 139.23ms (137.40, 140.94) 1.17+-
evaluation +0.42%, 0.05s invariant (0.46d, 0.40p, 0.10std) +1.10%, 0.01s invariant (0.99d, 0.09p, 0.01std) +1.88%, 15.1ms invariant (0.87d, 0.15p, 14.01std) -1.09%, -0.1ms faster ✓ (-1.62d, 0.01p, 0.06std) +0.50%, 0.7ms invariant (0.32d, 0.56p, 2.02std)
CairoMakie 11.27s (11.12, 11.68) 0.20+- 1.18s (1.14, 1.23) 0.03+- 209.59ms (204.18, 218.77) 5.31+- 11.08ms (10.87, 11.40) 0.16+- 6.66ms (6.56, 6.78) 0.08+-
master 11.29s (11.06, 11.53) 0.17+- 1.17s (1.13, 1.22) 0.03+- 221.36ms (215.45, 230.45) 4.88+- 11.15ms (11.03, 11.51) 0.17+- 6.66ms (6.45, 6.86) 0.14+-
evaluation -0.19%, -0.02s invariant (-0.12d, 0.83p, 0.19std) +0.85%, 0.01s invariant (0.32d, 0.55p, 0.03std) -5.62%, -11.77ms faster✅ (-2.31d, 0.00p, 5.10std) -0.64%, -0.07ms invariant (-0.44d, 0.43p, 0.16std) -0.05%, -0.0ms invariant (-0.03d, 0.96p, 0.11std)
WGLMakie 14.26s (14.04, 14.44) 0.17+- 1.35s (1.30, 1.39) 0.03+- 12.69s (12.40, 13.29) 0.31+- 16.77ms (15.15, 17.82) 1.02+- 1.22s (1.12, 1.27) 0.05+-
master 14.33s (14.07, 14.54) 0.16+- 1.38s (1.33, 1.45) 0.04+- 12.62s (12.28, 12.99) 0.22+- 16.90ms (15.42, 18.28) 0.89+- 1.19s (1.17, 1.21) 0.01+-
evaluation -0.48%, -0.07s invariant (-0.42d, 0.45p, 0.16std) -1.97%, -0.03s invariant (-0.70d, 0.22p, 0.04std) +0.54%, 0.07s invariant (0.26d, 0.64p, 0.26std) -0.76%, -0.13ms invariant (-0.13d, 0.81p, 0.96std) +2.69%, 0.03s invariant (0.89d, 0.14p, 0.03std)

@asinghvi17
Copy link
Member

asinghvi17 commented Apr 21, 2023

It would be interesting to have a "datashader axis", which has the capability to datashade any plot plotted into it. It could theoretically do this by just consuming the atomic plot elements (lines + linesegments + scatter) and rasterizing them. Surfacelikes could potentially be disallowed/ignored.

This might be useful to visualize densely connected graph clusters and similar strategies. We could then also

@SimonDanisch SimonDanisch marked this pull request as ready for review May 10, 2023 19:47
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file is huge, should be mp4

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(and you can also embed videos such that they auto-play and auto-loop)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

docs/examples/plotting_functions/datashader.md Outdated Show resolved Hide resolved
docs/examples/plotting_functions/datashader.md Outdated Show resolved Hide resolved
docs/examples/plotting_functions/datashader.md Outdated Show resolved Hide resolved
src/basic_recipes/datashader.jl Show resolved Hide resolved
src/basic_recipes/datashader.jl Outdated Show resolved Hide resolved
src/basic_recipes/datashader.jl Outdated Show resolved Hide resolved
src/basic_recipes/datashader.jl Outdated Show resolved Hide resolved
src/basic_recipes/datashader.jl Outdated Show resolved Hide resolved
Comment on lines 273 to 274
colormap = theme(scene, :colormap),
colorrange = automatic
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO the colorrange calculation should be slightly changed here, so that we don't have to fiddle with colormaps. Let the colorrange be (1, max) instead of (0, max), then set lowclip = :transparent, or :black, or what have you. This should also make the colormap a lot more exact!

@github-actions
Copy link
Contributor

Missing reference images

Found 1 new images without existing references.
Upload new reference images before merging this PR.

@github-actions
Copy link
Contributor

Missing reference images

Found 1 new images without existing references.
Upload new reference images before merging this PR.

@github-actions
Copy link
Contributor

Missing reference images

Found 1 new images without existing references.
Upload new reference images before merging this PR.

@SimonDanisch
Copy link
Member

@jkrumbiegel I'm going to merge this when this goes green... It's still not perfect, but I cant invest much more time into this.
I added a warning, that its still experimental!

@jkrumbiegel
Copy link
Member Author

Yeah I'm ok with that. Not much more movement on it, maybe once people can actually use it.

@jkrumbiegel
Copy link
Member Author

Should this work together with the new CategoricalColormap?

@SimonDanisch
Copy link
Member

Should this work together with the new CategoricalColormap?

For now, I only implemented proper legend support for categorical datashader:
image
And normal colorbar for non categorical datashader:
image

@jkrumbiegel
Copy link
Member Author

What's the legend element with the gradient? Did you add a new legend element kind?

@SimonDanisch
Copy link
Member

It's a poly marker with per vertex color ;)

@jkrumbiegel
Copy link
Member Author

It's a poly marker with per vertex color ;)

While it's kind of cool, I think I'd still change those to normal solid colors. It's the color that tells you the category, not the alpha.

@SimonDanisch SimonDanisch merged commit dd418b1 into master Sep 20, 2023
13 checks passed
@SimonDanisch SimonDanisch deleted the jk/datashader branch September 20, 2023 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants