Integration with Elixir and Explorer

[josevalim]

Hi @jonmmease, thank you for the amazing work on this project!

We have been working on bringing data and machine learning to Elixir. We currently have a dataframe tools called Explorer (built on top of Polars), and we are discussing integrating it with VegaFusion. If that's ok, we have some questions about making this possible.

It seems the best way to get started is by implementing the basic quickstart feature with pre_transform_vega. Elixir's starting point is a dataframe and a VegaLite spec. So we use Polars "to_ipc/to_arrow" to get the underlying Arrow representation and vl-convert to convert the VegaLite spec to Vega. We feed that into the relevant vegafusion-runtime API to get an updated Vega spec. Question: what is the simplest way to create a VegaFusionRuntime, assuming I have a pointer to an ArrowSchema already in memory?

In the future, we will want to accelerate interactive charts, but we can postpone it for now. However, I do have some questions about it. Imagine I am using the movies dataset (the same used in the pre-transform notebooks) loaded into Polars for the interactive functionality. For interactivity, would we need to keep a copy of the original movies dataset in memory? Or do we keep a copy of the transformed dataset instead?

Thank you!

[jonmmease]

Hi @josevalim, thanks for reaching out. I'm a big fan of Elixir and would love to make sure VegaFusion is flexible enough for what you need.

So we use Polars "to_ipc/to_arrow" to get the underlying Arrow representation and vl-convert to convert the VegaLite spec to Vega. We feed that into the relevant vegafusion-runtime API to get an updated Vega spec.

Just to orient myself here, are you already wrapping the polars and vl-convert-rs Rust crates in Elixir?

Question: what is the simplest way to create a VegaFusionRuntime, assuming I have a pointer to an ArrowSchema already in memory?

The VegaFusion runtime performs caching, but it's logically stateless, so you only need to create a single instance of it. In the Python API, we group the VegaFusionRuntime and a tokio thread pool here

vegafusion/vegafusion-python-embed/src/lib.rs

Lines 134 to 139 in 007bd44

	#[pyclass]
	struct PyVegaFusionRuntime {
	runtime: Arc<VegaFusionRuntime>,
	tokio_runtime_connection: Arc<Runtime>,
	tokio_runtime_current_thread: Arc<Runtime>,
	}

and then in Python we create a global instance of the Python wrapper for this here

vegafusion/python/vegafusion/vegafusion/runtime.py

Line 772 in 007bd44

runtime = VegaFusionRuntime(64, psutil.virtual_memory().total // 2, psutil.cpu_count())

In the future, we will want to accelerate interactive charts, but we can postpone it for now. However, I do have some questions about it. Imagine I am using the movies dataset (the same used in the pre-transform notebooks) loaded into Polars for the interactive functionality. For interactivity, would we need to keep a copy of the original movies dataset in memory? Or do we keep a copy of the transformed dataset instead?

For interactive charts, take a look at the ChartState struct.

vegafusion/vegafusion-runtime/src/task_graph/runtime.rs

Line 907 in 007bd44

pub struct ChartState {

The idea is that a ChartState stores the current interactive state of a chart on the server, and it provides the communication plan necessary for the client to decide what callbacks must be installed with the Vega renderer in order to maintain interactivity. This is how the Vega-Altair JupyterChart works when the "vegafusion" data transformer is enabled. See https://github.com/vega/altair/blob/main/altair/jupyter/jupyter_chart.py.

Because the VegaFusionRuntime's cache is free to evict cache entries during the lifetime of the interactive chart, the original datasets do need to be accessible throughout the lifetime of the interactive chart. This doesn't need to be an independent copy of the dataset, it can be a reference to the dataset. e.g. in Python this is usually the pandas DataFrame passed to the Altair chart.

Let me know if you have more specific questions. It might be easiest for me to answer if you look at the vegafusion-python-embed crate and let me know what's still unclear.

[josevalim]

Just to orient myself here, are you already wrapping the polars and vl-convert-rs Rust crates in Elixir?

We have one for polars and we will implement one for vl-convert.

The VegaFusion runtime performs caching, but it's logically stateless, so you only need to create a single instance of it.

I assume the cache is still mutable though, is this correct? And I assume I can read and write to the cache concurrently under several threads or should I place a mutex/rwlock around it?

I think those pointers are enough for getting started. Thanks for the feedback and the nice words!

[jonmmease]

Yes, the cache is mutable. Its internals are protected by async_lock primitives where needed, so you can call it's associated functions concurrently.

Good luck, and I'm interested to hear how it goes!

[jonmmease]

Hi @josevalim, I was curious to check in on whether you moved forward with this integration. I'm working working on cleaning up various aspects of VegaFusion for a coming 2.0 release, and on improving the documentation for lower-level use cases like this, so any feedback you have would be useful.

[josevalim]

Unfortunately I was not able to prioritize this, so there was no progress for now, sorry! And thanks for checking!

[jonmmease]

no worries, thanks for getting back to me!