Skip to content

Commit

Permalink
Merge pull request #827 from saulpw/develop
Browse files Browse the repository at this point in the history
Prepare for v2.1 release
  • Loading branch information
saulpw authored Dec 7, 2020
2 parents 1132abf + e69959d commit 9c83fe4
Show file tree
Hide file tree
Showing 112 changed files with 2,403 additions and 750 deletions.
123 changes: 123 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,128 @@
# VisiData version history

# v2.1 (2020-12-06)

- [add] add bulk rows and cols leave cursor on first added (like add singles)
- [add] add colname input to `addcol-new`
- [aggregators] add mode and stdev to aggregator options (thanks @jsvine for PR #754)
- [api] add options.unset()
- [columns] add hidden 'keycol' to **ColumnsSheet** (thanks @geekscrapy for feature request #768)
- [cli] support running as `python -m visidata` (thanks @abitrolly for PR #785)
- [cli] add `#!vd -p` as first line of `.vdj` for executable vd script
- [cli] allow `=` in `.vd` replay parameters
- [clipboard] clipboard commands now require some selected rows #681
- [commands] add unset-option command bound to `d` on OptionsSheet #733
- [config] `--config=''` now ignores visidatarc (thanks @rswgnu for feature request #777)
- [defer] commit changes, even if no deferred changes
- [deprecated] add traceback warnings for deprecated calls (thanks @ajkerrigan for PR #724)
- [display] add sort indication #582
- [display] show ellipsis on left side with non-zero hoffset (thanks @frosencrantz for feature request #751)
- [expr] allow column attributes as variables (thanks @frosencrantz for feature request #659)
- [freq] change `numeric_binning` back to False by default
- [input] Shift+Arrow within `edit-cell` to move cursor and re-enter edit mode
- [loaders http] have automatic API pagination (thanks @geekscrapy for feature request #480)
- [loaders json] improve loading speedup 50% (thanks @lxcode for investigating and pointing this out #765)
- this makes JSON saving non-deterministic in Python 3.6, as the order of fields output is dependent on the order within the dict
- (this is the default behaviour for dicts in Python 3.7+)
- [loaders json] try loading as jsonl before json (inverted)
- jsonl is a streamable format, so this way it doesn't have to wait for the entire contents to be loaded before failing to parse as json and then trying to parse as jsonl
- fixes api loading with http so that contents of each response are added as they happen
- unfurl toplevel lists
- functionally now jsonl and json are identical
- [loaders json] try parsing `options.json_indent` as int (thanks @frosencrantz for the bug report #753)
this means json output can't be indented with a number. this seems like an uncommon use case
- [loaders json] skip lines starting with `#`
- [loaders pdf] `options.pdf_tables` to parse tables from pdf with tabular
- [loaders sqlite] use rowid to update and delete rows
- note that this will not work with WITHOUT ROWID sqlite tables
- [loaders xlsx] add active column (thanks @kbd for feature request #726)
- [loaders zip] add extract-file, extract-selected, extract-file-to, extract-selected-to commands
- [macros] add improved macro system (thanks @bob-u for feature request #755)
- `m` (`macro-record`) begins recording macro; `m` prompts for keystroke, and completes recording
- macro can then be executed everytime provided keystroke is used, will override existing keybinding
- `gm` opens an index of all existing macros, can be directly viewed with `Enter` and then modified with `Ctrl+S`
- macros will run command on current row, column sheet
- remove deprecated `z Ctrl+D` older iteration of macro system
- [regex] use capture names for column names, if available, in `capture-col` (thanks @tsibley for PR #808)
- allows for pre-determining friendlier column names, saving a renaming step later
- [save] `g Ctrl+S` is `save-sheets-selected` on **IndexSheet**
- new command allows some or all sheets on an **IndexSheet** to be saved (and not the sheets on the sheet stack)
- [saver] add fixed-width saver (uses col.width)
- [saver sqlite] ensureLoaded when saving sheets to sqlite db
- [search] `search-next` and `searchr-next` are now bound to n and N (was `next-search` and `search-prev`)
- [select] differentiate select-equal- and select-exact- (thanks @geekscrapy for feature request #734)
- previous select-equal- matched type value
- now select-equal- matches display value
- add `z,` and `gz,` bindings for select-exact-cell/-row
- [sheets] sorting on **SheetsSheet** now does not sort **SheetsSheet** itself. (thanks @klartext and @geekscrapy for bug reports #761 #518)
- [status] use `color_working` for progress indicator (thanks @geekscrapy for feature request #804)
- [types] add floatsi parser (sponsored feature by @anjakefala #661)
- floatsi type now parses SI strings (like 2.3M)
- use `z%` to set column type to floatsi

## Bugfixes

- [api] expose visidata.view (thanks @alekibango for bug report #732)
- [color] use `color_column_sep` for sep chars (thanks @geekscrapy for bug report)
- [defer] frozen columns should not be deferred (thanks @frosencrantz for bug report #786)
- [dir] fix commit-sheet and delete-row on DirSheet
- [draw] fix display for off-screen cursor with multiline rows
- [expr] remove duplicate tabbing suggestions (thanks @geekscrapy for bug report #747)
- [expr] never include computing column (thanks @geekscrapy for bug report #756)
- only checks for self-reference; 2+ cycles still raises RecursionException
- caches are now for each cell, instead of for each row
- [freeze] freeze-sheet with errors should replace with null
- [loaders frictionless] assume JSON if no format (thanks scls19fr for bug report #803)
- from https://specs.frictionlessdata.io/data-resource/#data-location):
- a consumer of resource object MAY assume if no format or mediatype property is provided that the data is JSON and attempt to process it as such.

- [loaders hdf5] misc bugfixes to hdf5 dataset loading (thanks @amotl for PR #728)
- [loaders jsonl] fix copy-rows
- [loaders pandas] support loading Python objects directly (thanks @ajkerrigan for PR #816 and scls19fr for bug report #798)
- [loaders pandas] ensure all column names are strings (thanks @ajkerrigan for PR #816 and scls19fr for bug report #800)
- [loaders pandas] build frequency table using a copy of the source (thanks @ajkerrigan for PR #816 and scls19fr for bug report #802)
- [loaders sqlite] fix commit-sheet
- [loaders sqlite] fix commit deletes
- [loaders xlsx] only reload Workbook sheets to avoid error (thanks @aborruso for bug report #797)
- [loaders vdj] fix add-row
- [man] fix warnings with manpage (thanks @jsvine for the bug report #718)
- [movement] fix scroll-cells (thanks @jsvine for bug report #762)
- [numeric binning] perform degenerate binning when number of bins greater than number of values
- (instead of when greater than width of bins)
- [numeric binning] if width of bins is 1, fallback to degenerate binning
- [numeric binning] degenerate binning should resemble non-numeric binning (thanks @setop for bug report #791)
- [options] fix `confirm_overwrite` in batch mode
- fix `-y` to set `confirm_overwrite` to False (means, no confirmation necessary for overwrite)
- make `confirm()` always fail in batch mode
- make `confirm_overwrite` a sheet-specific option
- [plugins] only reload **Plugins Sheet** if not already loaded
- [replay] move to replay context after getting sheet (thanks @rswgnu for bug report #796)
- [replay] do not push replaying .vd on sheet stack (thanks @rswgnu for bug report #795)
- [scroll] zj/zk do nothing in single-line mode (thanks @jsvine for suggestion)
- [shell] empty stdin to avoid hanging process (thanks @frosencrantz for bug report #752)
- [status] handle missing attributes in `disp_rstatus_fmt` (thanks @geekscrapy for bug report #764)
- [tabulate] fix savers to save in their own format (thanks @frosencrantz for bug report #723)
- [typing] fix indefinite hang for typing (thanks @lxcode for issue #794)
- [windows] add Ctrl+M as alias for Ctrl+J #741 (thanks @bob-u for bug report #741)
- [windows man] package man/vd.txt as a fallback for when man is not available on os (thanks @bob-u for bug report #745)

## Plugins
- add conll loader to **PluginsSheet** (thanks @polm)
- remove livesearch
- add clickhouse loader

## Commands
- if `options.some_selected_rows` is True, `setcol-expr`, `setcol-iter`, `setcol-subst`, `setcol-subst`, `setcol-subst-all` will return all rows, if none selected

## API
- [columns] add Column.visibleWidth
- [open] additionally search for `open_filetype` within the vd scope
- [select] rename `someSelectedRows` to `onlySelectedRows`
- [select] add new `someSelectedRows` and `options.some_selected_rows` (thanks maufdez for feature request #767)
- if options is True, and no rows are selected, `someSelectedRows` will return all rows
- [status] allow non-hashable status msgs by deduping based on stringified contents
- [isNumeric] isNumeric is part of vdobj

# v2.0.1 (2020-10-13)

## Bugfixes
Expand Down
1 change: 1 addition & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
don't be a dick
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ Some examples of great bug reports:

## Submitting Source Code

Check out the [Plugin Authors Guide](https://visidata.org/docs/api) for an overview of the API.
Code in `plugins/` or `visidata/loaders/` is welcome, as long as it is useful to someone and safe for everyone.
Updates or additions to the core code should be proposed via an [Github Issue](https://github.com/saulpw/visidata/issues/new/choose) before submitting a PR.

Expand Down
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
include README.md
include LICENSE.gpl3
include visidata/man/vd.1
include visidata/man/vd.txt
74 changes: 26 additions & 48 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,82 +1,55 @@
[![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)

# VisiData v2.0.1 [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable)
# VisiData v2.1 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/develop.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/develop) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)

A terminal interface for exploring and arranging tabular data.

![Frequency table](http://visidata.org/freq-move-row.gif)

## Dependencies
VisiData supports tsv, csv, sqlite, json, xlsx (Excel), hdf5, and [many other formats](https://visidata.org/formats).

- Linux, OS/X or Windows
- Python 3.6+
- python-dateutil
- other modules may be required for opening particular data sources
- see [requirements.txt](https://github.com/saulpw/visidata/blob/stable/requirements.txt) or the [supported sources](https://visidata.org/formats) in the vd manpage
## Platform requirements

## Getting started
- Linux, OS/X, or Windows (with WSL)
- Python 3.6+
- additional Python modules are required for certain formats and sources

### Installation
## Install

Each package contains the full loader suite but differs in which loader dependencies will get installed by default.
To install the latest release from PyPi:

The base VisiData package concerns loaders whose dependencies are covered by the Python3 standard library.
pip3 install visidata

Base loaders: tsv, csv, json, sqlite, and fixed width text.
To install the cutting edge `develop` branch (no warranty expressed or implied):

|Platform |Package Manager | Command | Out-of-box Loaders |
|-------------------|----------------------------------------|----------------------------------------------|----------------------|
|all |[pip3](https://visidata.org/install#pip3) | `pip3 install visidata` | Base |
|all |[conda](https://visidata.org/install#conda) | `conda install --channel conda-forge visidata` | Base, http, html, xls(x) |
|MacOS |[Homebrew](https://visidata.org/install#brew) | `brew install saulpw/vd/visidata` | Base, http, html, xls(x) |
|Linux (Debian/Ubuntu) |[apt](https://visidata.org/install#apt) | [full instructions](https://visidata.org/install#apt) | Base, http, html, xls(x) |
|Linux (Debian/Ubuntu) |[dpkg](https://visidata.org/install#dpkg) | [full instructions](https://visidata.org/install#dpkg) | Base, http, html, xls(x) |
|Windows |[WSL](https://visidata.org/install#wsl) | Windows is not directly supported (use WSL) | N/A |
|all |[github](https://visidata.org/install#git) | `pip3 install git+https://github.com/saulpw/visidata.git@stable` | Base |
|Linux (NixOS)|[nix](https://visidata.org/install#nix)| `nix-env -i visidata`|Base, yaml, xls(x), hdf5, html, pandas, shp |
pip3 install git+https://github.com/saulpw/visidata.git@develop

Please see [/install](https://visidata.org/install) for detailed instructions, additional information, and troubleshooting.
See [visidata.org/install](https://visidata.org/install) for detailed instructions for all available platforms and package managers.

### Usage

$ vd [<options>] <input> ...
$ <command> | vd [<options>]

VisiData supports tsv, csv, xlsx, hdf5, sqlite, json and more (see the [list of supported sources](https://visidata.org/formats)).
$ vd <input>
$ <command> | vd

Use `-f <filetype>` to force a particular filetype.
Press `Ctrl+Q` to quit at any time.

Hundreds of other commands and options are also available; see the documentation.

### Documentation

* [VisiData documentation](https://visidata.org/docs)
* [Plugin Author's Guide and API Reference](https://visidata.org/docs/api)
* [Quick reference](https://visidata.org/man) (available within `vd` with `Ctrl+H`), which has a list of commands and options.
* [Intro to VisiData Tutorial](https://jsvine.github.io/intro-to-visidata/) by [Jeremy Singer-Vine](https://www.jsvine.com/)
* Quick reference: `Ctrl+H` within `vd` will open the [man page](https://visidata.org/man), which has a list of all commands and options.
* [/docs](https://visidata.org/docs) contains a collection of howto recipes.

### Help and Support

If you have a question, issue, or suggestion regarding VisiData, please [create an issue on Github](https://github.com/saulpw/visidata/issues) or chat with us at #visidata on [freenode.net](https://webchat.freenode.net/).

Here are some concrete ways you can help make VisiData even more awesome:

* Write a blogpost (or tweet or whatever) about a VisiData command or feature you use frequently, and share it with us!
* Expand VisiData to support .xyz proprietary data format. Creating a loader [is really straightforward](https://www.visidata.org/docs/api/loaders.html).
* Create and maintain [new installation packages](https://github.com/saulpw/visidata/labels/packaging).
* Acknowledge the realities of late-stage capitalism and [give regular old money](https://www.patreon.com/saulpw).

## Other applications within the VisiData ecosystem

The core interface paradigm--rows and columns--can be used to create efficient terminal workflows with a minimum of effort for almost any application. These have been prototyped as proof of this concept:

- [vgit](https://github.com/saulpw/visidata/tree/stable/plugins/vgit): a git interface
- [vsh](https://github.com/saulpw/vsh): a collection of utilities like `vping` and `vtop`.
- [vdgalcon](https://github.com/saulpw/vdgalcon): a port of the classic game [Galactic Conquest](https://www.galcon.com)

Other workflows can also be created as separate apps using the visidata module. These apps can be very small and provide a lot of functionality; for example, see the included [viewtsv](https://visidata.org/docs/viewtsv).
If you use VisiData regularly, please [support me on Patreon](https://www.patreon.com/saulpw)!

## License

VisiData, including the main `vd` application, addons, loaders, and other code in this repository, is available for use and redistribution under GPLv3.
Code in the `stable` branch of this repository, including the main `vd` application, loaders, and plugins, is available for use and redistribution under GPLv3.

## Credits

Expand All @@ -85,3 +58,8 @@ VisiData is conceived and developed by Saul Pwanson `<vd@saul.pw>`.
Anja Kefala `<anja.kefala@gmail.com>` maintains the documentation and packages for all platforms.

Many thanks to numerous other [contributors](https://visidata.org/credits/), and to those wonderful users who provide feedback, for helping to make VisiData the awesome tool that it is.

[1.1]: http://i.imgur.com/tXSoThF.png
[1]: http://www.twitter.com/VisiData

<!-- Please don't remove this: Grab your social icons from https://github.com/carlsednaoui/gitsocial -->
95 changes: 60 additions & 35 deletions dev/ROADMAP
Original file line number Diff line number Diff line change
@@ -1,35 +1,60 @@
# Roadmap to 2.0

## API

The primary goal of 2.0 is a stable, documented Python API that can be used to create an broad ecosystem of loaders, plugins, and apps.

The Book of VisiData will describe the api and all functions and objects available to plugins and command execstrs.

## Split panes

+ horizontal split
- one pane can be dependent on the other; e.g. move cursor in one, see preview update in other
+ use for choosing aggregators, command help, other choose()

- DirSheet file preview #309

## add jsonl as more robust system format
+ add .vdj format (visidata cmdlog as jsonl)
+ migrate plugins.tsv to be jsonl format

## Minor features, refactors, cleanup

- 'modified' indicator on status bar, also enables quitguard
- remove all asserts; replace with error() on case-by-case basis

- [options] allow list or dict of enum values for default
- [options] for options that have been renamed, add aliases for compatibility
- e.g. `tsv_delimiter` and `tsv_row_delimiter` become 'proper' options with 'delimiter', 'delim', and 'd' becoming aliases

# Features

- Memory sheet; can give things names and use in expressions.
- 'show' commands are copied there by default.

- [columns] add helpstr for all specialized-sheet columns
# Roadmap for 2.x

1. holdovers from 2.0
- [canvas] API docs
- [options] option enums
- [options] user-defined option aliases
- [splitpane] File preview in directory view
- [defermods] 'modified since last save' indicator on status bar
- [scroll cell] zh/zl Left hand side of a cell with content hidden doesn't show ellipsis #751

2. Persistence
- input history #736 #468
- key indexes for better joining and lookups

3. Interface discoverability for commands #247 #742
- clickable menu canvas
- more clickable affordances all around
- clickable motd
- possibly add popup modals

4. more expressive expressions
- Memory sheet; can give things names and use in expressions. #392
- access column values from execstr #655
- shortcut name for current column #659

5. better asynchronicity
- async thread pool
- streaming architecture #366 #656

6. Loaders/Savers
- frictionless saver #237
- RSS reader #157
- toml loader #735
- HexSheet for unknown/binary files #548
- .ods loader (LibreOffice/OpenOffice spreadsheet) #473

- jsonl load and save round-trip (minimal diff) #429

## other features

a) generate non-terminal graphs (ggplot)
b) automatic reload into time series
c) intra-cell coloring (for search results, markup)

# major plugin projects

1. fully operational SQL viewer/editor

- #282: Select starting table in postgres from command-line
- #579: [Postgres] Allow inserting / deleting rows
- #522: [postgres] parms in options
- #586: SQL query data
- #727: [postgres] Transaction error when viewing table
- #729: Integrate generic SQL loader

2. web scraper

- #480: HTTP API pagination loader
- #465: Ability to load from the contents of a cell.
- #505: [html] Provide way to access non-table elements
Loading

0 comments on commit 9c83fe4

Please sign in to comment.