Use MakieReferenceImages repository

[jkrumbiegel]

Try to run CI with reference images downloaded from https://github.com/MakieOrg/MakieReferenceImages. If this works, in the next step one could set up automatic PRs against that repository after CI runs.

[MakieBot]

Compile Times benchmark

Note, that these numbers may fluctuate on the CI servers, so take them with a grain of salt. All benchmark results are based on the mean time and negative percent mean faster than the base branch. Note, that GLMakie + WGLMakie run on an emulated GPU, so the runtime benchmark is much slower. Results are from running:

using_time = @ctime using Backend
# Compile time
create_time = @ctime fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @ctime Makie.colorbuffer(display(fig))
# Runtime
create_time = @benchmark fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @benchmark Makie.colorbuffer(display(fig))

	using	create	display	create	display
GLMakie	3.67s (3.59, 3.71) 0.05+-	427.29ms (402.43, 461.97) 19.01+-	487.76ms (470.43, 508.40) 12.11+-	7.69ms (7.47, 8.32) 0.29+-	26.16ms (25.74, 26.48) 0.25+-
master	3.71s (3.62, 3.88) 0.09+-	386.44ms (331.06, 437.90) 43.90+-	507.16ms (480.46, 561.90) 27.31+-	7.63ms (7.31, 8.00) 0.27+-	26.20ms (25.93, 26.51) 0.22+-
evaluation	1.01x invariant, -0.04s (-0.56d, 0.32p, 0.07std)	0.90x noisy🤷‍♀️, 40.85ms (1.21d, 0.05p, 31.46std)	1.04x invariant, -19.4ms (-0.92d, 0.12p, 19.71std)	0.99x invariant, 0.06ms (0.23d, 0.68p, 0.28std)	1.00x invariant, -0.04ms (-0.17d, 0.76p, 0.24std)
CairoMakie	3.04s (3.02, 3.09) 0.02+-	319.46ms (316.22, 321.38) 1.74+-	141.90ms (140.54, 143.85) 1.06+-	7.15ms (7.11, 7.25) 0.05+-	608.59μs (604.56, 614.90) 3.60+-
master	3.01s (3.00, 3.02) 0.01+-	318.91ms (317.30, 320.67) 1.28+-	141.65ms (140.18, 142.24) 0.77+-	7.14ms (7.02, 7.27) 0.08+-	608.87μs (605.11, 613.36) 3.65+-
evaluation	0.99x slower X, 0.03s (1.62d, 0.02p, 0.02std)	1.00x invariant, 0.55ms (0.36d, 0.52p, 1.51std)	1.00x invariant, 0.25ms (0.27d, 0.62p, 0.92std)	1.00x invariant, 0.01ms (0.14d, 0.80p, 0.07std)	1.00x invariant, -0.28μs (-0.08d, 0.89p, 3.62std)
WGLMakie	3.85s (3.77, 4.05) 0.09+-	322.96ms (314.36, 332.16) 6.33+-	8.74s (8.58, 8.90) 0.12+-	9.29ms (9.01, 9.61) 0.23+-	69.91ms (67.90, 78.32) 3.74+-
master	3.80s (3.73, 3.86) 0.04+-	322.46ms (317.04, 334.34) 6.65+-	8.67s (8.56, 8.76) 0.07+-	9.44ms (9.16, 10.50) 0.47+-	69.39ms (67.47, 72.93) 2.13+-
evaluation	0.99x invariant, 0.05s (0.61d, 0.28p, 0.07std)	1.00x invariant, 0.5ms (0.08d, 0.89p, 6.49std)	0.99x invariant, 0.07s (0.71d, 0.21p, 0.09std)	1.02x invariant, -0.15ms (-0.41d, 0.46p, 0.35std)	0.99x invariant, 0.52ms (0.17d, 0.76p, 2.94std)

[jkrumbiegel]

We are going to wait with this until 0.20 is out

[SimonDanisch]

Decided to leave images in release artifacts, since managing the second github repository is too complicated.