Feedback on installing requests and lxml by default with VisiData

[anjakefala]

Pros is that they will come with every installation of VisiData, regardless if a package mantainer includes them or not.

Cons are that they would increase the size and complexity of the default installation, which might be undesireable if you do not use these features.

requests is used to load data from a url. e.g. vd https://visidata.org/usage.tsv. lxml is used to load html tables.

[rcaputo]

I voted none of the above because there are tools dedicated to fetching HTML (example: curl) and extracting data from it into something visidata already understands (example: pup).

Since data sources refresh at their own rates (sometimes never), I also prefer to decouple downloading data from processing and visualizing it. Possibly also a side effect of growing up when bandwidth was precious. Bong, bong, bong, screeech.

In use cases where I might want to redownload data every time, it's easy enough to script a pipeline that ends in vd.

[geekscrapy]

I voted none, as I found it extremely refreshing as I recently had to use vd on a remote appliance (which had a very limited py install) over ssh. Copied over the source and ran it, no dramas. No reqs issues at all.
Totally agree with @rcaputo on the "other tools to do this" pov.

[saulpw]

Yeah the only difference would be that pip install visidata would pull in more things by default. For example openpyxl isn't required by VisiData for anything but reading .xlsx files, but we install it anyway because it's small itself and people often (?) want to view Excel files without any additional hassle. I feel the same way about .html pages, it's super nice to just toss a wikipedia page at vd and have it get the tables out for me. Hence the poll.

[rcaputo]

Despite my opinion I wouldn't hate visidata if it included more things I don't (currently) use.

Duct taping pipelines together is fun, but so is finding out that vd already does something I just learned I needed.

[geekscrapy]

I may not be following correctly, but my usecase is that I couldn't install any additional reqs, apart from what is already vendored into vd. No connectivity and the appliance is locked down. So no pip installs or setup.py for me
In that scenario it helped a tonne as I couldn't drag back the large CSV files to my local host

[anjakefala]

I couldn't install any additional reqs

Yes, I can confirm that this will continue to work! You would not have access to those features that rely on those dependencies but base VisiData (other than the date type) would work.

[geekscrapy]

Ha thanks for clarifying (again 🙃) but yea, you reminded me that the only thing that caught me was the date thing!

[ryanprior]

I wrote requests + lxml, because I think the user experience degrades when you can't vd https://some-page-with-tables and have it just work. Since I see Visidata as an end-user workflow tool first, and a framework/library second, I think pip install visidata should give you a robust UX (at perhaps some cost in terms of bloat) while a "visidata-core" package could be made available for people who want the minimum, because they're using it as a framework or a library or otherwise want a simple install.

[ryanprior]

How much is "all?" Is it a lot by typical application standards, where you get megabytes of Electron cruft before the first line of application logic? Or just a lot by lean CLI tool standards? Does vd lazy-load dependencies or does installing more optional deps increase start-up time?

I think it would be cool if pip install visidata; vd some-obscure-format.file would just work: while everybody then risks downloading some KiB (MiB?) of libs they would personally never use, the one obscure thing each different person does use would be ready to go, which is so nice. If something doesn't work, how many people will think "oh I need to install a loader!" vs "oh shucks, vd doesn't support that." On the other hand, choosing some cutoff for uncommon loaders is also reasonable.

[saulpw]

Because "all" includes pandas and now arrow, it's getting up into 500MB territory I would estimate. And since we're doing vdsql, should we include all those databases? With bigquery and its dependency (and others--Ibis supports at least a dozen DBs), we're easily at 1GB or more.

[rcaputo]

500-1000MB seems excessive to me, but YMMV.

A compromise might be to download plugins and dependencies on the fly. It's almost like "just works" but with increased latency and security heebie-jeebies.

[ryanprior]

Yeah that's way outta bounds.

Spacemacs has a feature where if I load, say, a Python source file and I don't have the Python layer installed, it'll prompt me whether I want to install it just-in-time or not. That way, installing Spacemacs doesn't pull in a GiB of dependencies, but you avoid the mistaken impression that it lacks support for things that actually are covered.

Can we create vd loader stubs that trigger a message in the UI when you try to load something that's supported but not installed? "VisiData is missing the 'bigquery' package to load this data." etc

[geekscrapy]

Maybe a mapping between the file type being loaded and a loader plugin could be held within the vd repo itself?