Discuss if / how to put computational outputs into content blocks (notes, warnings, etc)

[choldgraf]

This has been discussed elsewhere (jupyter-book/jupyter-book#700, executablebooks/sphinx-exercise#18 for example) so moving here to have a single place to discuss this topic.

Current situation

Currently, executable code must exist in the form of a {code-cell} at the top level of a markdown file, or in the cell of a Jupyter Notebook. This is because we want a 1-to-1 mapping of text files <--> ipynb file via Jupytext, and since Jupyter Notebooks have a flat, top-level list of code cells, that is the only pattern that we support.

The problem

The problem that others have noted is that you often want to embed the results of running code inside other parts of your content. The most common example that has come up is to put code cells / outputs inside of an admonition. For example:

:::{note}
Some note text.

```{code-cell}
print("this is a code cell")
```
:::

The above pattern is not possible currently, because {code-cell} needs to be a top-level item, mimicking the structure of a Jupyter Notebook (which has no concept of code cells nested in other cells).

So one question we'd need to answer to support this is: how would the above code block be translated into an ipynb file? Right now I do not believe that it is possible with any existing tool.

A few potential solutions

Here are some ideas I can think of that we could discuss

1. Make it possible to nest code cells via cell metadata. In our Jupytext converter, we could support cell-level metadata like admonition: note, admonition-body: Some body text, admonition-footer, and when converted to markdown it would use this metadata to wrap the resulting code-cell in a {note} admonition.
   • The challenge with this is that it is very edge-casey...we're basically special-casing some cell metadata that would break down as soon as people wanted anything more complex, and it's also clunky to ask people to do complex configuration via cell metadata
2. Make it easier to inject code cells and their outputs into a page. Right now we have glue, a python function that tags a variable with a name for insertion. Others have pointed out this feels too cumberson, and is also python-specific. Perhaps making it easier to insert code cells / outputs from other pages will make this a workable solution.
   • The challenge with this is that we'd still require a two-step process for people to insert outputs etc. So maybe would still be too-cumbersome
   • See Pasting notebook cells / inputs / outputs into content MyST-NB#265 for some discussion.
3. Allow for in-page execution that doesn't map onto notebook cells. For example, we could introduce MyST-NB-specific syntax that would execute in the context of a Sphinx build, but would not directly translate to an ipynb file (it would just be copied as raw text or something).
   • For example, something like {execute}`print("hi")` would be run when the book was built, but would be meaningless in the context of a Jupyter interface.
   • The challenge with this is that we are currently using nbclient for our execution, and this doesn't support anything like what I describe.
   • We could consider using something like papermill and writing a post-execution hook that would run these blocks of code after executing all of the cells on the page.
   • This might be possible in nbclient if we had something like Function hooks when executing cells / notebooks? jupyter/nbclient#81

• Provide a way for extensions to hack this behavior - I guess the last thing we could do is to figure out how an extension like sphinx-exercise could work around this behavior (maybe by adopting its own special cell metadata?) and then document how other extensions could do the same thing.
  • The challenge with this is that it's also very edge-casey and may not be possible anyway...

I don't think any of these solutions feels obviously correct to me. Personally if the machinery were there to make 3 possible, I think that's the one I'd prefer since it is closest to what RMarkdown does, and people seem to quite like that feature.

Maybe others have ideas for supporting this through other means - I would love to hear your thoughts.

[chrisjsewell]

Allow for in-page execution that doesn't map onto notebook cells.

My thought is this is an all or nothing option: the files either map 1:1 markdown <-> notebook, or they are not associated with notebooks at all.
You can't have parts of the code that don't work in the notebook context, that would be a complete mess.
If you want to go with this option, we should just use jupyter-sphinx or something similar. It's a valid option, but note we will have to drop notebook specific functionality like Binder and Colab, plus a new/adapted caching system (as opposed to jupyter-cache), plus plus users would no longer be able to use pre-executed notebooks

(2) is the better option IMO

[choldgraf]

Yeah, I agree with the main drawback of option 3 - we're basically introducing two kinds of computational behavior depending on whether a file is run in a myst-nb context or a jupyter book context. Also agreed that we're basically talking about jupyter-shpinx (another way I'd describe option 2 is essentially "allow jupyter sphinx and myst-nb to share a kernel).

That said I still wonder if it is the lesser-of-all-evils, partially because this is already a mental map that people coming from the R community will have and that has proven to be very popular.

However, if we can find a pattern that follows 2 and that is both simple in UX and implementation, I agree that this would be the preferable option.

It would be helpful to hear from @jstac and the QuantEcon team as this seems to be a particularly important topic for them.

[akhmerov]

Also linking #143 with the same question being discussed. I think the main relevant point from that discussion is that it would be possible to support code cells nested anywhere, if jupytext conversion was based on myst-parser as well as sphinxcontrib-rst2myst and not only markdown-it-py, but this has unreasonably high implementation costs.

Out of the options above, I think (2) (a.k.a. executablebooks/MyST-NB#265) is indeed the most reasonable since it's completely general, and doesn't break user assumptions about execution order etc.

[chrisjsewell]

it would be possible to support code cells nested anywhere, if jupytext conversion was based on sphinxcontrib-rst2myst and not markdown-it-py

As I've pointed out in that issue, this is just not true, because you would still need markdown-it-py (+myst-parser) to convert to the doctree in the first place

[akhmerov]

Sorry, I fixed the description.

[chrisjsewell]

Yeh no worries, I just wanted to be very clear that sphinx-tomyst (formerly sphinxcontrib-rst2myst) is not in any way a solution to this. It acts on a defined subset of doctree nodes, and is not just generally applicable to all possible sphinx nodes (e.g. those created by extensions). It is meant as a helper tool for people who want to convert their rST file to MyST, but should definitely not be used as a core part of the build process.

(If you want to understand more, I suggest having a look at the source code
https://github.com/executablebooks/sphinx-tomyst/blob/71297ac5d0e6ff702de728b302981a4d87602754/sphinx_tomyst/writers/translator.py#L24)

[akhmerov]

Thanks, that clarified the limitations of sphinx-tomyst.

[akhmerov]

Another remark that I think is useful here is the meaning of 1-1 mapping myst ↔ ipynb. This, to the best of my understanding is a shorthand to the requirement that the following two sequences of actions produce the same result:

1. convert all myst to ipynb with jupytext
2. run all notebooks
3. build jb with execution turned off
4. inspect the resulting book as well as the notebooks

and

1. build jb from myst with execution via jupyter-cache
2. inspect the resulting book and the generated notebooks

This is more stringent than only requiring bidirectional conversion myst ↔ ipynb with jupytext. If that was the only constraint, then I believe it would be possible to support nested code cells with a lower complexity than dragging the complete stack into jupytext.

[chrisjsewell]

Huh? There's literally app.add_directive("code-cell", CodeCell)

Yeh and now look at CodeCell

https://github.com/executablebooks/MyST-NB/blob/446cf0d269a0906eacc1b88b418c7bce40ee4cee/myst_nb/__init__.py#L400

[chrisjsewell]

try to nest code-cell directives, and observe unexpected failures. This is actually what started this discussion: jupyter-book/jupyter-book#700, executablebooks/sphinx-exercise#18

Also both of these issues reference old versions of myst-nb, before the specific warning and link to documentation put in place

[akhmerov]

Oh wow. They certainly look a lot like directives.

[chrisjsewell]

At the end of the day, you clearly have a very specific use case / viewpoint on what you want to achieve. This is fine, but I don't get why you don't just implement all this in jupyter-sphinx?

[akhmerov]

Once again, this discussion isn't about my use case at all, and my personal educational resources are all working perfectly fine without nested code.

Still, to address your proposal: I would be happy to make jupyter-sphinx play more nicely as a stand-in replacement of myst-nb execution model, but #152 is limiting myst-nb interoperability.

[mmcky]

Sorry to jump in later in the discussion on this.

A comments for sphinx-exercise:

Would it be a reasonable assumption that code-cell blocks contained in admonitions are independent to the rest of the execution environment of that page? I think this could a reasonable assumption for sphinx-exercise:solution but probably not for note or other admonition types.

From the sphinx-exercise viewpoint we could compile a solutions notebook that is a companion document (document.ipynb, document_solutions.ipynb) and all code cells in the solutions are added to that companion notebook which would then:

1. build companion notebook of _solutions
2. run both documents
3. retrieve results for code-cell in solution block from _solutions notebook.

Alternative Build Approach:

Another alternative which is closely related to original Option 2 would be to compile during the build process:

1. a content/markup file, and
2. an execution file

during the build process. The code file would then contain all code blocks/code-cell as found in the document from top to bottom (nested or not). The content / markup file would then get compiled by sphinx pulling in outputs from the code file (after execution). A hash of the code-cell could be used as a linking id to merge the outputs with inputs? So the key assumption would be all code-cell blocks are run in a linear top to bottom fashion as they appear in the md document (i.e. like a notebook).

The downside as I see it from this approach is it would mix sphinx.ast with execution (if we used the ast to extract code-cells to produce a code-only notebook). The current approach is simpler with 1. conversion, 2. execution, 3. compile via sphinx.