Why does execution happen on env-get-outdated?

[akhmerov]

Hi! This is mainly a question to @chrisjsewell about the current implementation. Looking at how myst-nb hooks into sphinx, I see:

    app.connect("config-inited", validate_config_values)
    app.connect("builder-inited", static_path)
    app.connect("builder-inited", set_valid_execution_paths)
    app.connect("builder-inited", set_up_execution_data)
    app.connect("builder-inited", set_render_priority)
    app.connect("env-purge-doc", remove_execution_data)
    app.connect("env-get-outdated", update_execution_cache)
    app.connect("config-inited", add_exclude_patterns)
    app.connect("config-inited", update_togglebutton_classes)
    app.connect("env-updated", save_glue_cache)
    app.connect("config-inited", add_nb_custom_formats)
    app.connect("env-updated", load_ipywidgets_js)

I'd like to understand better why update_execution_cache, which executes all notebooks, is connected to "env-get-outdated", which is supposed to only determine which files need work.

The docs say "You can return a list of docnames to re-read in addition to these", while we're doing something a lot more mutating: executing the notebooks. The docstring of update_execution_cache doesn't really clarify why this processing has to happen at that step.

I can imagine a situation where such treatment of the sphinx pipeline will cause undesirable behavior. For example imagine I implement custom logic that determines that some notebooks should be rerun due to a dependency version change, external data being updated, or anything alike. IIUC applying that custom logic after myst-nb has registered itself as an extension would result in an incorrect output since the execution of the notebooks wouldn't be triggered.

[chrisjsewell]

which executes all notebooks

But that's the point, it doesn't execute all notebooks, it only executes the ones that have changed. This is stated in the docstring

which is supposed to only determine which files need work.

No that's not the case at all, they are callbacks within which you can implement whatever code you want.

IIUC applying that custom logic after myst-nb has registered itself as an extension would result in an incorrect output since the execution of the notebooks wouldn't be triggered

Just add a callback with a higher priority and return extra notebooks you want to be re-executed:

event.connect("env-get-outdated", my_callback, 499) 😄

[choldgraf]

It's a little bit unclear to me what you're getting at @akhmerov - are you suggesting that the event should be triggered at a different point in the build chain and trying to understand why not?

It seems like your use-case of "my notebooks have a different criteria for when they should be re-executed than jupyter-cache uses" would be solved by just setting execution to off and manually implementation your own execution logic as a part of your build. I feel like that level of custom requirement would also suggest "power-user" status, and thus we could expect that user to be a bit more savvy at coding things up themselves.

I think the general approach of executablebooks shouldn't be to support every case of reproducibility etc, it should be to cover the most common ones, and to have escape hatches to make it possible for users to hack their own solutions if they have particular constraints that EBP tools can't cover.

That said, it does seem like it's common for readers to want to bundle dependencies with their books. One thing that we could do is implement a logic like "users can list another collection of 'dependency files'. At build time if any of those files has changed, then it re-triggers execution of all notebooks".

[akhmerov]

are you suggesting that the event should be triggered at a different point in the build chain and trying to understand why not?

No, I don't know of a place in the build chain that seems better suited for execution. I do see other places that superficially seem similarly suited, which means that I am overlooking some part of the ideas behind myst-nb and/or sphinx. I am interested in this because I would like to contribute more constructively, and without proposing something unrealistic, rather than implementing some hypothetical advanced workflow. That's the only motivation for the question.

---

As an aside, I'm 💯 on extra build dependencies. I expect that a lot of larger books will seek to factor out common functionality (e.g. applying the plot styling) into a separate module to avoid copypaste.

[chrisjsewell]

sphinx/myst-nb should not be handling notebook execution or cacheing, it should be almost completely modular; people should be able to re-execute their notebooks outside of the documentation build, and the execution module should be able to decide how (multiple) notebooks are executed: parallel-processed, run inside docker containers, run on a HPC, ...
This is the point of jupyter-cache, and in the future execution will be increasingly defered to that https://github.com/executablebooks/jupyter-book/issues/947

[akhmerov]

Sure, I used "execute" as a shorthand to update_execution_cache. That still leaves the freedom of choosing when that happens in myst-nb. Or do you mean that myst-nb will eventually stop interacting with jupyter-cache and assume that all inputs are up to date?

[chrisjsewell]

Or do you mean that myst-nb will eventually stop interacting with jupyter-cache and assume that all inputs are up to date?

No, just there will be a more tightly defined execution interface: these are some notebook files that "may" need re-execution, this is some configuration, tell me when you're done.
The idea of forcing re-execution, due to changes in assets/dependencies is definitely desirable though