options changes, persistent input history, .vds format, and more

This release adds new features and changes some existing behavior. Please see the CHANGELOG for the complete list of changes and bugfixes.

1. The interpretation of "global" vs. "sheet-specific" CLI options has changed.

In v2.0 and v2.1, options given on the command-line were "sheet-specific" and only applied to files on the command-line, and -g (--global) had to be specified to make them apply to sheets created after launch. Now, options are "global" by default (affecting all sheets which don't have a sheet-specific override), and -n indicates that subsequent CLI options are sheet-specific. The -g option is now basically a no-op (though it can be given after -n (--nonglobal) to return to the default behavior after specifying some sheet-specific options). This fixes a few things that users (including myself) were consistently misinterpreting or getting frustrated with, and hopefully does not affect anyone adversely.

2. New feature: persistent input history

By popular demand, VisiData can now store user-provided input values so they persist between sessions. This is currently disabled by default, as it may have surprising privacy or security implications in some environments. Enable persistent input history by adding the following to visidatarc:

options.input_history='previnputs'

3. New type: floatlocale

The VisiData float type uses the Python builtin float, which only parses numbers in "international" notation. To parse locale-specific numbers (which use e.g. a decimal comma instead of a decimal point), there is now the floatlocale type, which parses numbers based on the LC_NUMERIC environment variable (which must be set before starting vd). A separate type was created because locale-based parsing was measured as 20x slower than the base float type. The type-floatlocale and type-floatlocale-selected (on ColumnsSheet) commands were added without keybindings; users who need this functionality will probably want to add the following to their visidatarc:

bindkey('%', 'type-floatlocale')
bindkey('g%', 'type-floatlocale-selected')

4. Option change: Fancy Chooser now disabled by default

The "fancy chooser", a tabular menu with helpstrings for choosing e.g. aggregators and jointypes, is now disabled by default. Choosing an aggregator or jointype now (again) uses the status/input line, and requires entering the value manually. To invoke the fancy chooser, use Ctrl+X. If the fancy chooser is preferred by default, set options.fancy_chooser=True.

5. New format: .vds (VisiData Sheet)

This new single-file format can store typed data and column information from multiple sheets. Use g Ctrl+S (save-all or save-selected) to invoke the multisheet saver, and then specify a filename with a .vds extension. When you load the file into vd, the Column typing and formatting will be the same!

6. New loader (and saver): geojson

Thanks @dracos for PR #876 implementing geojson loading and saving!

7. Several other improvements and bugfixes

v2.1.1

This release may be a small one, but it contains a handful of meaningful bugfixes that we wanted to get out the door.

Happy New Year, everyone!

- [macros] allow macro interfaces to be longnames (thanks @frosencrantz #787)
- [save] better default save filename for url sheets (thanks @geekscrapy #824)

Bugfixes

- [cmdlog] record column, sheet, and row info for open-cell
- [cmdlog] catch case of 'override' sheet for set-option
- [expr-col] `curcol` now works for multiple invocations (thanks @geekscrapy #659)
- [loaders postgres] account for postgres_schema when rendering Postgres tables (thanks @jdormit for PR #852)
- [loaders url] fail unknown URL scheme (thanks @geekscrapy for PR #84)
- [pyobj] fix Pyobj Sheets for lists (thanks @brookskindle #843)
- [pipe] handle broken pipes gracefully (thanks @robdmc #851)
- [scroll] fix issue with jagged scrolling down (thanks @uoee #832)
- [sort] fix bug where total progress in sorting is (100 * # of columns to sort) (thanks @cwarden)

v2.1

v2.1

This release fixes several bugs in 2.0 and fills in some of the gaps. Please see the CHANGELOG for the complete list of changes and bugfixes.

1. options.some_selected_rows

Some commands, like setcol-expr (g=) operate on the selected rows. Before 2.0, if no rows were selected, these commands would fall back to operating over all rows. This is intuitive and helpful, but can cause problems with non-interactive script replay: sometimes there are legitimately no selected rows, and so the command should not do anything (but also not fail).

So the behavior was changed in 2.0 to have reliable and deterministic results. It's reasonably easy to work around: select all rows with gs, repeat the command, and maybe unselect them all with gu. But this turned out to be ultimately disappointing for interactive use.

So, in 2.1, there is now an option some_selected_rows, which if True, causes these commands to operate on all rows, if none are selected:

• setcol-expr (g=)
• setcol-iter (gz=)
• setcol-subst (g*)
• setcol-subst-all (gz*)

This option applies only to interactive mode, and not to batch mode. In batch mode, commands that operate on selected rows will always use the list of selected rows, even if there are none (and so it won't do anything). Be careful about saving .vd scripts from sessions in which this option was set; the results when replaying in batch mode may not be as expected.

2. options.numeric_binning

For 2.0, ranged binning of numeric columns was implemented, and allowed to be disabled by setting options.numeric_binning to False.
(By default it was set to True.)

In 2.1, this option is now disabled by default, and must be enabled (whether globally or per-sheet) by setting it to True.

3. Shift+Arrow within edit-cell to move cursor and re-enter edit mode

By popular demand, pressing Shift+Arrow when editing a cell, will save the current value, move the cursor one row or column in the given direction, and re-enter edit mode. This makes it easier to enter data, either row-wise or column-wise.

This feature obsoletes options.cmd_after_edit, which was always a terrible hack. It has no effect now.

4. Expression can access column attributes

In 2.0, expressions could use attributes on the vd singleton or the current sheet object, and had sheet and row special variables to refer to the sheet and row objects themselves.

In 2.1, expressions can also use attributes of the expression column, and col can be used to refer to the column object directly. (So col.width and width both refer to the width of the expression column itself, if there are no columns named width.)

5. Differentiate select-equal- and select-exact-.

Previously, select-equal- (bound to ,) matched on the typed value. Now, select-equal- matches on the displayed value. New commands select-exact-cell/-row that match on typed value are bound to z, and gz, (the display value may be rounded or formatted with less precision, so using the typed value is "more precise".

(Thanks @geekscrapy for noticing)

6. New macro system

A new command macro-record (bound to m) starts recording commands for use in a macro. The same command stops recording and prompts for a keystroke to bind the macro to. When the given keystroke is pressed, the macro will be executed. All macros run their commands on the current (row, column, sheet), regardless of the cursor position when the macro was recorded.

The new command macro-sheet (bound to gm) opens an index of all current macros. A macro on this sheet can be viewed with Enter and then modified, saving changes with Ctrl+S

This feature obsoletes the older iteration of macro system, and removes the deprecated z Ctrl+D command.

(Thanks @bob-u for the suggestion)

7. Other Additions and Improvements

• Add mode and stdev aggregators.
• Add sort-order indicator in column header.
• Unset options with d on options sheet (also add options.unset() API function).
• .vdj scripts are now hashbangable. Put #!vd -p at the start of the .vdj file and watch it go! (Note: extension must still be .vdj for now.)
• The floatsi type (z%) can now parse SI strings like "2.3M" (thanks @anjakefala for sponsoring).
• Regex capture (;) will use capture names as column names, if available (thanks @tsibley).

8. Format-specific improvements

• [http] auto-paginate by continuing with the given links.next.url from the response.

• [json] 50% speedup for loading (thanks @lxcode).

• [pdf] options.pdf_tables to parse tables from pdf with tabular.

• [sqlite] Use internal rowid to update and delete rows (WITHOUT ROWID sqlite tables can no longer be modified).

• [zip] Add extract-file, extract-selected, extract-file-to, extract-selected-to commands (bound to x, gx, zx, and gzx on the ZipSheet).

• [fixed] Add fixed-width saver (uses col.width).

• [clickhouse] Add plugin for clickhouse loader.

VisiData v2.0.1

VisiData 2.0.1

major changes since 1.5.2 (last official release)

1. Plugin API
2. Undo/Redo
3. Deferred changes
4. Split Window

highlighted features added since 1.5.2

• [iota] add i family of commands (iota/increment)
• [unfurl-col] zM, which does row-wise expansion of iterables in a column
• [join] add merge jointype (thanks @sfranky for feature request #405)
• [freq/pivot] range binning for numeric columns
• [options] options parsing rewrite allows for per-sheet options
• [cli] stdout pipe/redirect
• more visibility for long values, with multi-line rows and offset scrolling
• add Alt/Esc as prefix for user keybindings
• [input] Ctrl+Y paste from cell clipboard and other improvements

new loaders since 1.5.2

• MIME (.eml) (2.-5)
• recutils (.rec) (2.-5)
• VCard (.vcf) (2.-5)
• IMAP (email)
• MySQL (thanks @p3k)
• PDF (text only)
• numPY (.npy/.npz)
• Unicode Separated Value (.usv)
• Excel 2007-2010 binary fomat (.xlsb) (suggested by @woutervdijke #246)
• feather, gbq, orc, parquet, pickle, sas, stata via pandas (thanks @khughitt for bug report #460)
• postgres support for connecting directly to rds (thanks @danielcynerio for PR #536)

For the full list, see below along with release notes from previous pre-releases, or view the CHANGELOG

changes since pre-release v2.-4

• Additions and Improvements

  • [aggregators] allow custom aggregators in plugins/visidatarc (thanks @geekscrapy for the feature request #651)
  • [loaders xlsx] automatically clean sheet name when saving; warn if sheet name changes (thanks @geekscrapy for the request #594)
  • [columns] unhide height attribute by default (thanks @frosencrantz for feature request #660)
  • add .vcf (VCard) loader
  • [sqlite] remove name of db from an SqliteSheet's name, only tablename
  • [syspaste] make syspaste- replayable and undoable (note that syspaste- value will be recorded in CommandLog)
  • [savers] many text saver filetypes via tabulate library (thanks @jsvine for original vdtabulate plugin)
  • [calc] ExprColumn no longer cached by default
  • [loaders rec] add new .rec file loader and multisheet saver (recutils)
  • [savers] implemented multisheet saver for both json and jsonl
  • [loaders eml] add new .eml file loader for email files with attachments

• Options

  • add options.incr_base (thanks @chocolateboy for the suggestion #647)
  • (former) options.force_valid_colnames renamed to options.clean_names
    • applies to Sheets and Columns now (thanks @geekscrapy for the request #594)
  • for --X=Y, do not replace - with _ in Y (thanks @forensicdave for bug report #657)
  • add options.default_height for visibility toggle (thanks @frosencrantz for feature request #660)
  • add support for -- option-ending option on CLI.
  • [input] default now fancy_chooser = True
    • when fancy_chooser enabled, aggregators and jointype are chosen with a ChoiceSheet.
    • s to select, Enter to choose current row, g Enter to choose selected rows, q to not choose any
  • numeric_binning is now True by default (enables numeric binning on PivotSheet and FreqTable for numeric columns

• Command changes and additions
  - (former) setcol-range (gz=) renamed to setcol-iter
  - (former) addcol-range-step (i) renamed to addcol-incr-step
  - (former) setcol-range (gi) renamed to setcol-incr
  - (former) addcol-range-step (zi) renamed to addcol-incr-step
  - (former) setcol-range-step (gzi) renamed to setcol-incr-step
  - add scroll-cells-* to scroll display of cells while remaining in a Column; bind to [g]z{hjkl}
  - (former) unbind z{hjkl} from scroll-col (thanks @geekscrapy for feature request #662)
  - add type-floatsi bound to z% (#661)
  - reload-selected now reloads all Sheets if none selected (thanks @geekscrapy for PR #685)
  - add customdate with fixed fmtstr for parsing (use z@ and input a fmtstr compatible with strptime (thanks @suntzuisafterU for feature request #677)

• Bugfixes

  • [DirSheet] use changed ext as filetype when loading files (thanks @frosencrantz for bug report #645)
  • [slide] several major improvements to column sliding; key column sliding now works (thanks much to @geekscrapy for bug hunting #640)
  • [open-row] Sheets Sheet should be removed from stack upon open-row (thanks @cwarden for the bug report)
  • [cli] re-add --version (thanks @mlawren for bug report #674)
  • [open-config] fix gO (thanks @geekscrapy for bug report #676)
  • [splitwin] handle swap case for single sheet (thanks @geekscrapy for bug report #679)
  • [loaders xlsx] handle None column names for all Sequence Sheet loaders (thanks @jsvine for bug report #680)
  • [settings] retrieve from cache for top sheet if obj is None (thanks @aborruso for the bug report #675)
  • [settings] check if option is set on specific sheet before falling back to override
  • [describe] have DescribeSheet use source column's sheet's options.null_value to calculate its null column (thanks @aborruso for the bug report #675)
  • [undo] ensure that undos for complex commands (like expand-cols) are set more frequently (thanks @frosencrantz for the bug report #668)
    • it is still possible to find race conditions if the user presses commands fast enough, however they should happen far less frequently
  • [vlen] fix numeric binning for vlen() (thanks @frosencrantz for bug report #690)
  • [pivot] fix pivot case where no aggregator is set
  • [pyobj] fix filtering for PyobjSheet
  • [DirSheet] fix sorting for directory column of DirSheet (thanks @frosencrantz for bug report #691)
  • [json] fix bug saving cells with nested date values (thanks @ajkerrigan for PR #709)
  • [input] fix Ctrl+W bug when erasing word at beginning of line
  • [plugins] import .visidata/plugins by default
  • [pandas] use a safer reset_index() to avoid losing data when updating a pandas index (thanks @ajkerrigan for PR #710)
  • [threads] disable add-row on ThreadsSheet (thanks @geekscrapy for bug report #713)

• deprecated

  • complete removal of status and statuses from deprecated (thanks @frosencrantz for bug report #621)
    • longnames are now open-status and show-status
  • remove cursorColIndex

• API and Interface

  • Sheet(*names, **kwargs) autojoins list of name parts
  • openSource(), aggregator(), and aggregators are now part of vdobj
  • set_option is now setOption
  • move isError to Column.isError
  • deprecate load_pyobj, now PyobjSheet
  • add .getall('foo_') which returns all options prefixed with 'foo_'; deprecated options('foo_')
  • nSelected is now nSelectedRows
  • make Column.width property, so setting is same as Column.setWidth
  • evalexpr is now evalExpr
  • format is now formatValue
  • SettableColumn.cache is now ._store
  • vdtype() is now vd.addType()
  • add addColumnAtCursor (thanks @geekscrapy for bug report #714)

• Plugins

  • update sparkline (thanks @layertwo #696)
  • plugin dependencies now install into plugins-deps (former plugin-deps)

• Dev niceties

  • Fully automate dev setup with Gitpod.io (thanks @ajkerrigan for PR #673)

v2.-4 pre-release

New Features (v2.-4)

1. reintegrate deferred mods; save with z Ctrl+S

First and foremost, the 'defermods' feature has been reintegrated into core VisiData.
It was added in v2.-1, but it caused some headaches, so we separated it out into a plugin for v2.-2, along with vls for the DirSheet.
Then we found that we really missed having the functionality right in vd alongside everything else, and not having to reach for a specialized tool.
@anjakefala got to the bottom of our headaches and did the heavy lifting to make sure that everything works properly now.
So now the DirSheet and SqliteSheets are once again able to 'defer' their modifications and deletions, until saved to the source with z Ctrl+S. Ctrl+S saves the sheet to a regular data file as usual.

See also the defermods page.

Thanks to @anjakefala for her diligence and hard work on this!

2. unfurl a column (unfurl-col bound to zM)

As suggested by @jsvine, unfurl is a row-wise expansion of a column with containers.

• expand-col does a column-wise expansion: 1 column is added for each key/value in the containers (assumes mostly homogenous data)
• melt converts non-key columns into rows (on a new melted sheet)
• unfurl-col is kind of like a shortcut for an expand followed by a melt (demo on twitter)

In my mind, unfurl is conceptually closer to expand-col, but since it pushes a new sheet, putting it on a Shift+ command feels more natural.
The unfurl-col command feels like a focused melt--melting lists or dicts in the cursor column, instead of the individual values of the non-key columns as melt does.
So it's now bound to z Shift+M.

3. 'i'ncrement and family

Generating a range of numbers turns out to be surprisingly handy.

Previously, if you wanted to make a column with a list of numbers, you'd have to use za to create a column, gs to select all the rows (losing your previous selection) gz= (setcol-range
array languages, the 'increment' operator generates a sequence of integers from whole cloth.

In fact there was a version of this in the rownum plugin, and another version in @jsvine's unfurl plugin.
So we added equivalent commands to stock VisiData.
Because of the tradition of using i-like characters in APL notation ('ι' in APL and 'i.' in J), and the coincidence
that i has remained one of the few unbound keystrokes, I decided to develop this set of increment concepts, and bind them to the i family of commands.
See below for the new commands.

4. cli options improved; set sheet-specific options from CLI

We had a couple of open issues with command line options, and finally I bit the bullet and just wrote a custom cmdline args parser, instead of using Python's argparse. It was actually about the same amount of code, but now the syntax is more regular, more permissive, and can set sheet-specific options multiple times and differently for different sheets.
Thanks to @tsibley for patient testing and precise bug reports.

See /options#cli for details on how the command line interface works.

Also the visidata_dir option (default $HOME/.visidata) can now be set via the environment variable $VD_DIR.
Likewise config can be set via $VD_CONFIG.

5. fancy_chooser

Setting the fancy_chooser option to True enables a fancy selection menu for adding aggregators and choosing jointypes.
Help text is included for each item; press Enter to choose a single item, or select multiple rows and press g Enter, or q to abort.
This uses the splitpane functionality from the previous release.

6. column format strings

Python has multiple syntaxes for formatting numbers within strings: %.02f which used with the % format operator, and {:.02f} used with .format().
Which should you use for in the fmtstr attribute on the Columns sheet? A Twitter poll was evenly split, and @tsibley suggested ¿porque no los dos?](https://twitter.com/trs/status/1182390622507827202)
So as of v2.-4, you can use either format mechanism! If you want thousands separators though, you have to use {:,.02f}. See /blog/fmtstr for details.

new and improved loaders

• IMAP (email): open an imap:// url and it'll download all your email, which you can then save to .jsonl like I did

• basic MySQL loader (requires ..; thanks @ )

• basic PDF text extractor (requires pdfminer)

• improved loaders: csv hdf5 html http json jsonl pandas pcap postgres xls yaml

Thanks to @ajkerrigan for actively following up on the pandas and json bug reports, and all of his work in making them better.

new commands

• z Shift+M unfurl-col

• i addcol-range adds a column with integer values reflecting the current row order.

• gi setcol-range sets values in the current column for selected rows

• zi addcol-range-step adds a column with values

• gzi setcol-range-step sets values

• x (Canvas) resize-x-input sets x axis dimensions on a plot

• y (Canvas) resize-y-input sets y axis dimensions on a plot

• (unbound) select-error

• (unbound) select-error-col

• (unbound) mouse-enable and mouse-disable

• (unbound) guard-sheet sets quitguard on current sheet

lots more improvements and bugfixes

And of course there are tons of bugfixes and smaller improvements throughout. For a more complete list, see the CHANGELOG.

v1.5.2: a few bugfixes

We are starting off the New Year with a very small patch for a few bugs.

Briefly,

1. Fixed g*
   • thanks to both @jsvine and @layertwo for noticing and notifying us.
2. If suspended during editline (with Ctrl+Z), VisiData will now correctly resume in the editline.
3. Ctrl+W on an empty value in editline does not error out of the editline.

And that's it.

Happy new year, everyone!

v1.5.1: Bugfixes and user feedback

The initial motivation for this patch was to fix an idle performance regression. In the interim, though, we have received a bunch of feedback for improvements and a couple of feature requests.

So thanks for all of your ongoing engagement and for continuing to share VisiData with the world. We hope you enjoy this final release of 2018!

A brief overview:

1. Improvements to the pandas loader.

   • thanks to @nickray for the bughunting

2. When set to True options.bulk_select_clear bulk select commands (|, g|, ,, etc) will clear the previously selected rows.

   • the default is set to False and exhibits the current behaviour of retaining the already selected rows
   • thanks to @aborruso for the suggestion

3. Shift+O once again pushes the global options sheet. You can find the sheet options by pressing zO. gO now opens the .visidatarc which can be edited and saved from within the VisiData interface.

4. Sorting is now asynchronous, per user request. Please let us know how you find it.

   • thanks to @apnewberry for starting that conversation

5. gz^ now renames all visible columns to contents of selected rows and gz* substitutes over all visible columns

   • thanks, once again, to @aborruso for both suggestions

As usual, the complete list of changes can be found in the Changelog.

v1.5: plugins, saving errors, and bugfixes

For the most part, this is a bugfix release. A few other improvements are here as well:

1. Basic plugin support

   • ~/.visidata added to sys.path on startup
   • copy plugin to e.g. ~/.visidata/vdfoo.py
   • add import vdfoo to .visidatarc to activate

2. Error saving

   • for tsv and json, errors are saved as options.safe_error string (default #ERR)
   • if that option is an empty string, error message is saved instead

3. Mouse improvements

   • click now moves to column also
   • right-click to edit cell, rename column, or rename sheet, depending on where clicked.

4. xls/xlsx loaders now use options.header to determine number of rows used to set column names (default is 1).

See the CHANGELOG for a more complete list of bugfixes and changes.

v1.4: bugfixes and performance improvements

This is largely a bugfix and performance release. It was originally planned as a patch release (v1.3.2), but some features and interface changes crept in, so it is officially v1.4.

• [pyobj] visidata.view(obj) and visidata.view_pandas(df)
• [pandas] -f pandas loads file with pandas.read_<ext>
• [date] date minus date now gives float number of days instead of seconds
• [vdtui] internal colorizer API changed significantly
• [vdtui] timeout and redraw only if other threads active

As usual, a more complete list of changes and features is in the CHANGELOG.

See the ROADMAP for the planned future direction of VisiData.

v1.3.1: patch for aggregator issues

We found some issues with 1.3 (aggregators interacting with nulls/errors, primarily) that we didn't want to let sit until 1.4. So we fixed those issues and a couple others, and added a few 'minor' features. This patch version should be a definitive improvement over the base 1.3 version.

The complete list of changes is in the CHANGELOG. Here are the new options and features:

• new extend join type keeps the type of the first sheet, extending it with columns from the other sheets
• rename-sheet command (thanks to @jsvine for suggestion; what default keybinding should it have?)
• [DirSheet] add reload-rows (gz^R) to undo modifications on all selected rows
• remove all options.foo_is_null and add options.null_value
• options.save_errors (default True) to include errors when saving
• add options.json_indent for json pretty-printing