Data loaders?

[bzm3r]

Description

I am wondering how something like Observable Framework style "data loaders" fit into the design of Evidence.dev?

Put differently, what are the advantages of loading data only through SQL queries?

What problem would this solve?

It would allow a user to load data using a dataframe library like polars or dpylr, instead of DuckDB+SQL alone.

How should it work?

(I don't have good ideas on this yet, I am not even sure if the idea aligns well with Evidence's design/direction.)

[archiewood]

You can also load data outside of SQL eg via APIs if you build a custom data source. There are some examples (e.g. Stripe API) in https://github.com/archiewood/evidence-connector-stripe

[archiewood]

I guess it would be nice to just have the ability to drop a js (or even python) script into a source and have it output a parquet file in the right format for evidence to ingest

[benyn]

Yes, that would be nice. That's similar to how Observable Framework data loaders work. You write data loaders in any language, and Observable Framework will capture whatever's written to stdout, even console.log() statements. The output can be in any of the 17 supported formats (JSON, CSV, etc.) in Observable, but that also requires you to specify the file format when you use the file. For Evidence, I'd think it's just simpler to have every output file standardized as parquet.

[archiewood]

I've just published a javascript connector - this means anything you can run in javascript you can load into evidence without a new connector. It will handle all the conversion to parquet for you. All you need is to export a variable called data

https://github.com/archiewood/evidence-connector-js

sources/pokemon/pokedex.js

let url = 'https://pokeapi.co/api/v2/pokemon/'

const response = await fetch(url)
const json = await response.json()
const data = json.results

export { data }

index.md

```sql pokemon
select * from pokemon.pokedex
```

[benyn]

That was quick! Thanks Archie. Will give it a try.

[archiewood]

Did you try this out?