Plans for adding unzip-http support?

[frosencrantz]

@saulpw you have a new repo https://github.com/saulpw/unzip-http that looks like it would be great to add to VisiData or at least a new plugin.

I know I would find this useful, having recently tried to download a large zip data file, when it would have been useful to have just one file from the repo, without the full download.

[saulpw]

This would be quite feasible. I have some other plans for unzip_http in the near future involving Apache Arrow, and once those are in place, I could see adding more general http/.zip support to VisiData. Thanks for the suggestion, check back in a couple of months :)

[frosencrantz]

Looking forward to it! Thanks, Saul!

[frosencrantz]

@saulpw (and @anjakefala) It looks like you've already started to add this to Visidata! Wow! It works really fast to get the directory listing.

I don't typically pull data with zip files, but this actually makes it easy to quickly examine the contents!

If you plan on working on it more, I'll mention a couple of things I noticed when trying it out:

• Opening a file in the zip archive feels slow, and it would be nice if there was a percentage progress during loading
• For non-supported formats it would be useful if you could locally save the contents of a file from the archive.
• The ZipSheet doesn't have a file extension column like DirSheet does
• The timestamp is not set properly:

  File "visidata/loaders/archive.py", line 29, in <lambda>
    getter=lambda col, row: datetime.datetime(*row.date_time)),
TypeError: datetime.datetime() argument after * must be an iterable, not int

• For some files it fails with this error: Exception: Accept-Ranges header must be 'bytes' ('')

[saulpw]

Glad you like it, @frosencrantz! Progress should already be working in VisiData, while it's downloading. Unfortunately if the web server is slow there's not much we can report in the way of progress before the actual file starts streaming.

Similarly, for that last one, that's because the web server doesn't implement range requests to get partial bits of the file. So it's not possible to work in that case.

I did just implement the timestamp functionality and extractall(), so now you can use the extract- family of commands that already exists on the ZipSheet, bound to x (and zx, etc).

[anjakefala]

Haha, I love this energy you have with new features.

[frosencrantz]

Somehow this new feature is fun for looking at remote zip files. 😄 It is another new super feature of Visidata.
How else can someone do this as quickly without using unzip-http or Visidata?

@saulpw just a couple responses:

• I missed the extract commands, which covers everything I thought was missing. 👍
• On the loading progress, Visidata does show progress but it doesn't show the percentage progress. It seems like the zip directory has provided the size of the content and compressed content so it would be possible to report the percentage loaded so far. And it can take some time to load data, so a percentage progress would help report how it is proceeding.
• On the Accept-Ranges exception, should Visidata offer to download the zip file without using the optimisation? I'm not sure. It might be too expensive. Maybe the user should download it first.

[saulpw]

Hm, it shows loading progress for me as a percentage. Are you trying it with a big enough file? You might not see the percentage if it's just waiting on the web server, but the file transfer itself is small enough to get in one go.

Also, the extract commands are working for local .zip files, but they weren't working for remote .zip files; I only just implemented the required functionality last night. It will be on develop soon.

For the last one, it's definitely not worth it to auto-download a huge .zip file. Maybe an option could specify a max size to download, but it's not a priority right now. As you say, the feature is fun enough as it is :) I also don't think I've seen this feature anywhere else, which is weird because it's actually pretty easy to implement!

[frosencrantz]

I'm running from the latest develop branch, but maybe you have some unpublished changes? I see that there is progress, but not the percentage progress.

Here is what I'm seeing: https://asciinema.org/a/QLu6ec9m49qnWCPPcR6VnAjWQ
I quit before the larger table fully loads, but you can see that I'm not seeing any percentage progress.

This was the command from the demo:

vd 'https://files.digital.nhs.uk/publicationimport/pub18xxx/pub18273/nhs-staf-2015-healthcare-gp-csv.zip'

On the auto-download, that is reasonable.

My browser wants to just download zip files, which might be too much.

[saulpw]

Okay, I see that too with that use case. Can you file this as an issue so we don't forget about it?

[frosencrantz]

I've tried the latest head version and it has even more great improvements!:

• loading time has significantly improved.
  • The demo file from earlier in the thread, it would take a lot of time to even load some of the medium sized files, now I can load the largest file in a fraction of a minute
• I like that the columns better match the DirSheet columns.
  • I will use this as an opportunity to plug an old feature request to hide the directory column if there is only one directory [fdir] Default 'vd' command, hide the 'directory' column #1057

As you have asked, I've filed the new feature request for loading progress:

• [unzip-http] Loading percentage progress #1429

I did notice one issue:
The latest changes in develop now make it possible to load much larger files. The thing I noticed is that Visidata freezes up while reading the data. Example video:
https://asciinema.org/a/004aj3okZqwmjZkLUGQHYT7a6

In the demo I start loading a file from the zipfile. It shows the start of the sheet with a single column. Then I hit '^Tq' it takes more than 10seconds for Visidata to respond.

[frosencrantz]

I was trying this some more and found some stack traces: #1430