Executing cell in specific language

[lemeb]

As mentioned in #509, #507, and #450, it is now possible in Livy to execute Spark code in Python, Scala, R for any Livy session. That would be great if you want to write polyglot notebooks and, as it turns out, it is quite a low-hanging fruit.

A few notes:

• It uses an existing command line argument: %%spark run -l [language] for IPython kernels, or %%spark -l [language] for Sparkmagic kernels.
• All unit tests pass. I did not write any additional tests, though.
• I did not modify the documentation — or the example notebooks.

Here is a (slightly doctored for the sake of simplicity) working example. The json file is from here.

That was simple enough; now, for further, hard questions:

• Can you get proper syntax highlighting for each cell?
• How could we refactor the code such as you could have one Scala notebook and one Pyspark notebook working on the same session? (That is definitely doable.)

Anyway, thanks for all your hard work! Peace ✌️

[itamarst]

This is pretty cool. Thank you! To get this merged, we'd probably need:

1. Make sure there are some examples in the notebooks, just so people know it's a feature!
2. Some unit tests.

Not really sure if it's possible to do syntax highlighting per cell...

[lemeb]

thanks @itamarst! I'll try to have these two tasks done in the next few weeks when my thesis is done. :) (Right now, I'd had to boot sparkmagic out of my computers because it wasn't playing well with anaconda.)

Keep up the good work!

[snehab14]

Hi @lemeb, this is great!
Are you still working on getting this merged? This would definitely provide a lot of value.

[snehab14]

Why do we need to default this to the session kind? A statement without any kind attribute will use the session kind by default. Reference: https://livy.incubator.apache.org/docs/latest/rest-api.html#session-kind

[lemeb]

hi @devstein ! Is there still a need for this PR? I could work on finishing it if you want.

Best :)

[devstein]

Hi @lemeb yes! This would a great addition. Let me know if I can help. Most of the merge conflicts are likely caused by enforcing black #605

[shay1bz]

This feature was implemented also in #892 (independently) on the latest master branch.