Looking for collaborators to build "Visidata on the web"

[mukund109]

Hi all,

I've been using Visidata for a few weeks now and its my go-to tool for doing simple analytics on tabular data.

Its such an amazing tool, I desperately want a similar experience on the web. As such I've decided to build a visidata-like tool for the browser. I have the following goals in mind for this project:

• Make it easier to share tabular data (and different views of the data): Sharing of "sheets" should be as easy as sending someone a link that they can then open up in a browser. Uploading and exporting should also be super easy.
• Optimize for OLAP workloads on up to a few hundred GB of data: Since vd loads all rows into memory, it's unable to handle data that exceeds the size of memory. Analytical operations (like shift-f) on large datasets can be slow. This comes with certain advantages although I'd like "Visidata on the web" to be optimized for read-only analytical queries on small to medium sized data.
• Cater to both non-technical folks and power users: Visidata is a power-tool targeted at people who're familiar with the terminal and VIM keybindings, but its usefulness is universal. "Visidata on the web" will allow non-technical folks to derive basic utility out of the tool without overwhelming them. At the same time it will retain its use as a power-tool.

I've been playing around with a simple prototype for a few days.

So I was looking for people who'd like to collaborate on this project with me and see where it goes. I could use the help of a frontend developer (or someone willing to learn frontend development), since I don't know much about it (I come from a data science background).

I'm also looking for any thoughts or advice that people (Saul in particular) might have.

Also, I'm interested in knowing if there are tools out there that are similar to this. I feel like vd is different enough from BI tools (like Metabase) and data sharing tools (like Datasette) and spreadsheet tools (like Google Sheet) that there's a need for it out there. Let me know if you disagree.

[saulpw]

This is really cool, @mukund109! There are some really good ideas here and your prototype already looks pretty snpapy. I know some other people who might be interested in collaborating on this. Send me an email and we'll try to make it happen!

[mosherbrian]

@mukund109 It's funny -- I was just thinking how cool this concept would be and was searching online to see if there was a pyodide package to allow Visidata to run on the web when I ran across your message. It seems like you are thinking of something similar in concept at least. Very cool.

I could use the help of a frontend developer (or someone willing to learn frontend development)

I was a C++ developer in a past life, and although I wouldn't call myself a frontend developer, I have recently been spending quite a lot of my spare time learning html, css and vanilla JS so I can use them in my projects. I am not an expert, but would be interested to see whether I could be of assistance with something like this.

One thing I was thinking would be very cool is if an online Visidata could be embedded in Jupyter or Google Colab notebooks. Along with all of your good suggestions, that really seems like it would be a killer application for this.

If you are interested, drop me a note when you can. I would love to chat more about this.

[mukund109]

Hi all, been a while.

After a long time I got finally a chance to work on this project and managed to produce an MVP that I'm happy with.

https://tablehub.io/

Its got bare minimum features and some keybindings might be different. But have a look and let me know what you think.

[daviewales]

That's pretty cool! How hard would it be to support arbitrary data sets with o <paste CORS enabled URL to CSV>? (Might want to only load the first n rows)

[daviewales]

e.g. https://swsphn.github.io/nsw-covid-19-data/confirmed_cases_table1_location_agg.csv

[dufferzafar]

@mukund109 The site seems down now, do you have the repo open-sourced as well somewhere? would like to give it a shot!

[mukund109]

Hey @dufferzafar

The domain had expired but I recently renewed it. Here's the source code - https://github.com/mukund109/vow

[akaak]

hey @mukund109 Thanks for working on this project. I have tried the example datasets on your website https://tablehub.io/ and it looks good!

[paddymul]

Hello all! I have a somewhat similar table tool to visidata, but targeted at table/dataframe manipulation in the jupyter notebook as opposed to a terminal. Buckaroo I am looking at extracting the core dataframe table components from buckaroo (separating it from the managing widget state aspects) and shipping them as independent components.

I spoke with Saul about collaborating on this as a web frontend for visidata. A couple of points about what is possible, what is difficult, what is out of scope

• Using the tableviewer component of Buckaroo without the jupyter infrastructure, or the extra toolbars
  • Eminently doable. This is the next area of functionality I plan to build out.
• A static HTML file backed view of any table layout that visidata has
  • Probably the best first start. The core datastructures of Buckaroo are pretty simple and should be easy for visidata to export to. Here are the core typescript types
• A live server that shows the same dataframe view that terminal does. Very possible
• An interactive set of swappable frontend views and styles (similar to named pages in visidata).
  • Yes this is possible, you would probably want a different UI, but the full Buckaroo widget has concepts which enable this
• Providing similar keyboard navigation and functionality in Buckaroo backed by visidata compute on the web.
  • This is probably out of scope. But I would be open to talking about it.
  • A good first step would be plugging in a visidata calc backend to buckaroo. Buckaroo currently has pandas and polars backends.

With Buckaroo I have put a lot of work into wrapping the ag-grid table component and making it much easier to configure in a declarative way that doesn't require writing js.

I'm happy to answer any questions you all have and see if we can figure out how to collaborate.

[asteroidb612]

I've been looking into this from a different direction - building on top of a virtualized linux-in-the-browser project from webvm. I'm looking for some help from visidata community here: #2654