Client-side data source / upload component

[jpdenford]

Description

I'd like to create an evidence site which has a privacy respecting 'bring your own' data model:

• allows users select/drag & drop a file from their local machine
• transform the file (in my case I'd like to take a .mbox file, parse it and transform it into rows to match a pre-defined schema)
• add the data to client-side duckdb instance
• have the evidence queries run against the client-side duckdb dataset

Perhaps what I'm describing here can already be achieved using a custom component / component queries?
I only just discovered evidence and am really impressed by the project however I don't yet have a good understanding of the architecture (only gleaned from this issue) so hopefully what I'm suggesting makes sense.

Thanks

What problem would this solve?

Would allow using evidence to build dashboards which give people insight into their own data.

How should it work?

No response

[jpdenford]

I've just had another look through the existing discussions and this one sounds really ideal in terms of the interactivity aspect. Potentially worth keeping this discussion to capture the concept of client-side parsing / transform / insertion.

Thinking a little further through what I'm suggesting (and after reading a few more discussions containing evidence arch. info), I think the value add parts of a deployment, in the style I'm suggesting, are:
a. the ability for users to write their own queries against their data (I think this is already captured elsewhere and SqlConsole looks like a great place to start!!)
a. the pre-defined queries / graphs (which the 'domain expert' would create up front)
a. the code to parse / transform the source data file into a sql table (or tables?)

Data Upload
I can envision a scenario where a user would want to define a schema themselves and upload an arbitrary tabular formatted file which requires no transformation (e.g. csv, parquet). Personally I'm interested in the custom format / transforms for now.

How should it work

(already achievable I think) Create a new data source plugin type: schema-only or type: client-side which looks for files in the sources dir that define table schemas using a format like yaml (column names & their evidence types). The data source returns 0 rows.

Add a custom upload component which parses the data and then truncates & inserts it into the duckdb table/s. By default buildQuery seems to assume that the query is a 'select' (rather than an insert/update).

I'm realising that it may in fact be possible to do all this already with the right custom component!? I'll have a play when I get some time and see how I get on.