Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tick event #3948

Merged
merged 42 commits into from
Aug 9, 2024
Merged

Add tick event #3948

merged 42 commits into from
Aug 9, 2024

Conversation

ffreyer
Copy link
Collaborator

@ffreyer ffreyer commented Jun 8, 2024

Description

This pr adds events.tick which currently contains:

struct Tick
    state::TickState    # rendering state (paused, skipped, one-time or normal render)
    count::UInt64       # number of frames rendered 
    time::Float64       # since screen/backend event initialization
    delta_time::Float64 # between last and current frame
end

This is more verbose than just a delta time mostly because of how screen.render_tick is used in GLMakie. Updates to it are grouped with event polling since we have mouse position updates depend on it. So it triggers when

  • rendering, regardless of whether a frame is rendered or reused
  • rendering is paused, as not processing events would make the OS assume the program is not responding
  • plot and scene insertion for the same reason (these get filtered out)
  • calls to colorbuffer (i.e. save, record)

Depending on what causes the tick update a different state flag is passed. event_delta_time is always updated based on the time since the last tick. frame_delta_time is always based on the last renderloop event (paused/skipped/drawn) or colorbuffer event (i.e. a draw for save or record) and set to 0 when the current event is not one of those. So it's the time between the last and current frame, even if plot insertions gets between frames.

Currently the pr should be pretty functional, but I think we need to discuss what we want ticks to be on the user side.

  • Should we ignore ticks coming from plot insertions, i.e. only include ticks related to rendering? (Which would make event_delta_time redundant) (Currently does)
  • Should a paused renderloop still tick? (Currently counted)
  • Should we track more data in ticks, like frame count and time-since-start as suggested in Global figure time #3299 (comment) (How should we treat reused frames with frame count? Should record reset the time-since-start?) (Currently tracking frame count, time since start, time since last frame)
  • Should record overwrite the delta times in ticks to match the fps of the recording (rather than using the real time between rendered frames)? (Currently does not. My thought being that you can fairly easily use (tick.count - first_count) / fps to get the relevant delta_time.)

Supersedes #3299
Related: #3163

Type of change

  • New feature (non-breaking change which adds functionality)

Checklist

  • Added an entry in CHANGELOG.md (for new features and breaking changes)
  • Added or changed relevant sections in the documentation
  • Added unit tests for new algorithms, conversion methods, etc.
  • Added reference image tests for new plotting functions, recipes, visual options, etc.

@MakieBot
Copy link
Collaborator

MakieBot commented Jun 8, 2024

Compile Times benchmark

Note, that these numbers may fluctuate on the CI servers, so take them with a grain of salt. All benchmark results are based on the mean time and negative percent mean faster than the base branch. Note, that GLMakie + WGLMakie run on an emulated GPU, so the runtime benchmark is much slower. Results are from running:

using_time = @ctime using Backend
# Compile time
create_time = @ctime fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @ctime Makie.colorbuffer(display(fig))
# Runtime
create_time = @benchmark fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @benchmark Makie.colorbuffer(fig)
using create display create display
GLMakie 4.51s (4.40, 4.62) 0.07+- 111.32ms (106.12, 123.27) 6.38+- 562.13ms (544.56, 599.22) 19.62+- 8.99ms (8.58, 9.27) 0.26+- 26.42ms (26.16, 26.55) 0.13+-
master 4.49s (4.43, 4.56) 0.05+- 109.98ms (105.51, 115.83) 3.61+- 561.81ms (542.57, 597.56) 18.83+- 8.97ms (8.53, 9.54) 0.35+- 26.47ms (26.30, 26.74) 0.15+-
evaluation 1.00x invariant, 0.02s (0.31d, 0.57p, 0.06std) 0.99x invariant, 1.34ms (0.26d, 0.64p, 5.00std) 1.00x invariant, 0.32ms (0.02d, 0.98p, 19.23std) 1.00x invariant, 0.03ms (0.09d, 0.87p, 0.30std) 1.00x invariant, -0.05ms (-0.36d, 0.51p, 0.14std)
CairoMakie 4.01s (3.96, 4.06) 0.04+- 111.22ms (107.51, 114.39) 2.80+- 135.75ms (131.67, 141.18) 3.76+- 8.84ms (8.54, 9.16) 0.26+- 0.99ms (0.97, 1.00) 0.01+-
master 3.99s (3.93, 4.07) 0.06+- 107.85ms (105.06, 110.25) 1.82+- 134.68ms (127.87, 156.74) 9.94+- 8.96ms (8.31, 9.18) 0.30+- 976.54μs (968.87, 982.32) 4.91+-
evaluation 0.99x invariant, 0.02s (0.47d, 0.40p, 0.05std) 0.97x slower X, 3.37ms (1.42d, 0.02p, 2.31std) 0.99x invariant, 1.07ms (0.14d, 0.80p, 6.85std) 1.01x invariant, -0.13ms (-0.44d, 0.42p, 0.28std) 0.99x invariant, 0.01ms (1.13d, 0.07p, 0.01std)
WGLMakie 4.59s (4.48, 4.62) 0.05+- 106.80ms (104.88, 109.11) 1.42+- 9.13s (8.79, 9.27) 0.21+- 9.44ms (9.34, 9.52) 0.08+- 71.13ms (70.52, 71.58) 0.42+-
master 4.63s (4.53, 4.74) 0.07+- 106.72ms (104.59, 113.65) 3.20+- 9.19s (9.02, 9.41) 0.13+- 9.60ms (9.28, 10.57) 0.44+- 71.75ms (71.16, 72.12) 0.34+-
evaluation 1.01x invariant, -0.04s (-0.68d, 0.23p, 0.06std) 1.00x invariant, 0.08ms (0.03d, 0.95p, 2.31std) 1.01x invariant, -0.07s (-0.39d, 0.48p, 0.17std) 1.02x invariant, -0.16ms (-0.50d, 0.38p, 0.26std) 1.01x faster ✓, -0.62ms (-1.63d, 0.01p, 0.38std)

@jkrumbiegel
Copy link
Member

Closing #3163 would need using this to replace all time-dependent animations in Makie. I think that might just be the toggle switch animation, the textbox cursor animation and the axis reset timer after scrolling.

@ffreyer
Copy link
Collaborator Author

ffreyer commented Jun 10, 2024

Example from #3299 with this:

begin
    fig = Figure()

    axs = [Axis(fig[i, j]) for i in 1:2, j in 1:2]
    xlims = [[-1.2, 0], [0, 1.2]]
    ylims = [[0, 1.2], [-1.2, 0]]
    wavelengths = [
        5 8
        11 14
    ]
    ω = 0.3
    θs = range(0, 2π ; length = 1000)

    for i in 1:2 
        for j in 1:2
            ax = axs[i, j]
            xlims!(ax, xlims[j])
            ylims!(ax, ylims[i])
            w = wavelengths[i, j]

            ρs = @lift [sin(w *+ ω * $(events(fig).tick).count * 0.033)) for θ in θs]
            xx = @lift $ρs .* cos.(θs)
            yy = @lift $ρs .* sin.(θs)

            lines!(ax, xx, yy)
        end
    end
    
    fig
end

record(fig, "flower.mp4", 1:240) do i
end

It's a little awkward that you have to use tick.,count instead of tick.time for record here, since time is always real time atm. I'm still not sure if it should use fos based times in record. That would simplify this example a little more, but you couldn't use ticks for rendering stats during record anymore.

@jkrumbiegel
Copy link
Member

jkrumbiegel commented Jun 11, 2024

tick.time for record here, since time is always real time

Why not both if both can be useful. For the animations in record I would want a frame time and not the time that elapsed in reality.

Edit: Ah I see, the animations would then probably look wrong in live mode. Maybe a keyword to record could control which time base should be used? Sometimes people want to record stuff live I guess but most of the time I'd think not.

@ffreyer
Copy link
Collaborator Author

ffreyer commented Jun 11, 2024

I moved colorbuffer ticks to Makie record/save so they can synchronize to record framerates now. I'm not entirely sure if they are in the correct functions, but the tests I've written are passing at least.

The ticks produced with WGLMakie are a bit of a mess though. On screen initialization there tends to be a burst of them, and during record the normal render loop is also running, producing ticks. Idk if there is anything we can do about that, maybe @SimonDanisch can have a look?

@ffreyer
Copy link
Collaborator Author

ffreyer commented Jun 12, 2024

┌ Error: Cannot resolve @ref for md"[`TickState`](@ref)" in src/api.md.
│ - No docstring found in doc for binding `Makie.TickState`.
│ - No docstring found in doc for binding `Main.TickState`.

Does this not work with enums?

Also "Normals of a Cat" is failing due to changes in axis limits. Not idea where that is coming from. I don't see it locally, not even as a temporary state of the axis

@jkrumbiegel
Copy link
Member

Normals of a cat has been flaky on CI for me too. Haven't yet tried to figure out what the problem is.


For an interactive figure this will produce an animation synchronized with real time. Within `record` the tick times match up the set `framerate` such that the animation in the produced video matches up with real time.

The only exception here is WGLMakie which currently runs the render loop during recording. Because of that `RegularRenderTick` and `OneTimeRenderTick` will mix during `record`, with the former being based on real time and the latter based on video time. If you do not restrict to `OneTimeRenderTick` here the resulting video will run faster than real time with `tick.delta_time` and may jump with `tick.time` and `tick.count`.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably avoid this by eating up the regular render ticks during recording

cb = on(events(fig).tick, priority = typemax(Int)) do tick
    return Consume(tick.state != Makie.OneTimeRenderTick)
end
# record ...
off(cb)

but it's not clear to me how to do that safely. My first thought would be to have the creation of VideoStream start this and save end this, but that doesn't seem save to me. I.e. VideoStream being created doesn't necessarily mean only recording takes place and you could save another video after the first save...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added that to record and suggested doing this for the lower level functions in the docs for now.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I changed my mind and just attached the filtering to VideoStream and save/a finalizer. Mostly because otherwise the tick event order is a bit weird.

In the previous version the tick updates triggered just before colorbuffer(). So the order of operations was:

# record(foo, ...)
foo()
tick_update()
colorbuffer()

You'd still get updates coming from on(..., tick) before the render, but you couldn't use events(fig).tick to get the current frames time or count in foo(). I change things around a bit so that record effectively becomes:

tick_update()
foo()
colorbuffer()

so you can. This makes the filtering more important though, because the update for tick N actually happens at the end of frame N-1 in the recordframe!() call. (The first tick happens on VideoStream creation)

I also adjusted ticks so that the first tick is at count and time = 0.

@ffreyer
Copy link
Collaborator Author

ffreyer commented Jun 19, 2024

With the current state of this pr you can set up an animation like this for example:

angle = Observable(0.0)
pos = map(angle) do phi0
    [Point2f(cos(phi + phi0), sin(phi + phi0)) for phi in range(0, 2pi, 101)[1:end-1]]
end
f, a, p = scatter(pos, color = 1:100, strokewidth = 1, strokecolor = :black)
scatter!(pos[][1], marker = Circle, markersize = 12, color = :transparent, strokewidth = 2, strokecolor = :red)
on(events(f).tick) do tick
    angle[] = 0.5 * pi * tick.time
end
f

and you will see 1 revolution every 4s independently of (set and real) frame rate.

You can record this animation by just calling record:

record(f, "test.mp4", 1:120, framerate = 30) do _
    nothing
end

This will generate a new set of ticks starting at count = t = 0 with time steps of 1.0 / framerate = 0.0333.... So it will reset the animation to the time = 0 state, and then record one revolution, as 120 frames / 30fps = 4s.

If you then redisplay the figure, the interactive ticks will again be based on the renderloop. The first tick may have a long delta_time = current_time - time_before_record.

@ffreyer ffreyer marked this pull request as ready for review July 25, 2024 19:50
@ffreyer
Copy link
Collaborator Author

ffreyer commented Aug 8, 2024

Just noting - the way record ticks work might be incompatible with #3883

src/ffmpeg-util.jl Outdated Show resolved Hide resolved
@ffreyer ffreyer merged commit d7b5a11 into master Aug 9, 2024
18 checks passed
@ffreyer ffreyer deleted the ff/render_tick branch August 9, 2024 13:22
@ffreyer ffreyer mentioned this pull request Aug 9, 2024
5 tasks
@SimonDanisch SimonDanisch restored the ff/render_tick branch October 21, 2024 11:30
@SimonDanisch SimonDanisch deleted the ff/render_tick branch October 21, 2024 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Merged
Development

Successfully merging this pull request may close these issues.

4 participants