Merge pull request #857 from saulpw/develop

v2.1.1

CHANGELOG.md

@@ -1,5 +1,25 @@
 # VisiData version history
 
+# v2.1.1 (2021-01-03)
+
+    - [macros] allow macro interfaces to be longnames (thanks @frosencrantz #787)
+    - [save] better default save filename for url sheets (thanks @geekscrapy #824)
+
+## Bugfixes
+    - [cmdlog] record column, sheet, and row info for open-cell
+    - [cmdlog] catch case of 'override' sheet for set-option
+    - [expr-col] `curcol` now works for multiple invocations (thanks @geekscrapy #659)
+    - [loaders postgres] account for postgres_schema when rendering Postgres tables (thanks @jdormit for PR #852)
+    - [loaders url] fail unknown URL scheme (thanks @geekscrapy for PR #84)
+    - [pyobj] fix Pyobj Sheets for lists (thanks @brookskindle #843)
+    - [pipe] handle broken pipes gracefully (thanks @robdmc #851)
+    - [scroll] fix issue with jagged scrolling down (thanks @uoee #832)
+    - [sort] fix bug where total progress in sorting is (100 * # of columns to sort) (thanks @cwarden)
+
+## api
+    - format_field formats int(0) and float(0.0) as "0" (thanks @geekscrapy for PR #821)
+    - add TypedWrapper.__len__ (thanks @geekscrapy)
+
 # v2.1 (2020-12-06)
 
     - [add] add bulk rows and cols leave cursor on first added (like add singles)

README.md

@@ -1,5 +1,5 @@
 
-# VisiData v2.1 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/develop.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/develop) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)
+# VisiData v2.1.1 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/develop.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/develop) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)
 
 A terminal interface for exploring and arranging tabular data.
 

dev/history.jsonl

@@ -6,3 +6,5 @@
 {"date": "2020-05-13", "event": "changelog podcast released", "url": "https://changelog.com/podcast/394"}
 {"date": "2020-12-05", "event": "anja's tweet about visidata being a tool everyone should gets >100 likes", "url": "https://twitter.com/nevoitbien/status/1335398803604586498"}
 {"date": "2020-12-06", "event": "[HN] VisiData in 60 seconds", "url": "https://news.ycombinator.com/item?id=25322091"}
+{"date": "2020-12-05", "event": "[reddit] VisiData - a cool vim-like tool for examining and manipulating data", "url": "https://www.reddit.com/r/vim/comments/kf4wgb/visidata_a_cool_vimlike_tool_for_examining_and/"}
+{"date": "2020-12-06", "event": "RealPython tweets about VisiData", "url": "https://twitter.com/realpython/status/1339669258448547842"}

docs/columns.md

@@ -203,10 +203,13 @@ These variables and functions are available in the scope of an expression:
 - **`sheet`**: the current sheet (a TableSheet object)
 - **`col`**: the current column (as a Column object; use for Column metadata)
 - **`row`**: the current row (a Python object of the internal rowtype)
+- **curcol**: evaluate to the typed value of this row in the column that the cursor was on at the time that the expression column was added.
+- **cursorCol**: evaluate to the typed value of this row for the column the cursor is on. Changes as the cursor moves for `=`. Uses the column from the time the calculation was made for `g=`, `gz=`, and `z=`.
 
 Additional attributes can be added to sheets and columns.
 
-`col` deliberately returns a Column object, but any other Column object is interpreted as the value within that column for the same row.
+`col` deliberately returns a Column object, but any other Column object is interpreted as the value within that column for the same row. For example, both `curcol` and `cursorcol` return values, not the object itself.
+
 
 For example, this customizes addcol-expr to set the `curcol` attribute on the new ExprColumn to a snapshot of the current cursor column (at the time the expression column is added):
 
@@ -235,6 +238,19 @@ The following examples use the file [sample.tsv](https://raw.githubusercontent.c
 2. Set the type of the new derived column by pressing `@` (date).
 3. Type `^` followed by `Date` to rename the column to **Date**.
 
+**Question** I have a dataset with **Date** column that is missing a prefix of '2020-'. How do I add it to the **Date** column?
+
+When using `=`, and wanting to reference the current column, we recommend using `curcol`. When using `g=`, `gz=`, and `z=`, we recommend cursorCol. `=`, unlike the others, is dynamic and changes with adjustment of underlying values, which means it will change along with the movement of the cursor (tracked by `cursorCol`). `curcol` is a special attribute of a new **ExprColumn**, which remembers the cursorCol at the time of creation.
+
+1. Move the cursor to **Date**.
+2. Type `g=` followed by *f"2020-{cursorCol}"*.
+
+**Question** I have a dataset with **file names**. How do I create a new column with the **file names** lower cased?
+
+1. Move the cursor to **file names** column.
+2. Type `=` followed by **curcol.casefold()**.
+3. Move to the newly created column, and rename it with `^`, followed by the desired name.
+
 ---
 
 ## How to configure multiple columns

docs/index.md

@@ -37,6 +37,8 @@
     * How to perform operations on a subset of rows
     * How to filter rows
     * How to filter a random subset of rows
+    * How to select rows where the current column is not null
+    * How to select rows where the current column is null
     * How to move, copy and remove rows
     * How to sort rows
 * [Columns](/docs/columns)

docs/rows.md

@@ -65,6 +65,23 @@ The following example uses the file [sample.tsv](https://raw.githubusercontent.c
 
 ---
 
+## How to select rows where the current column is not null?
+
+'Null' cells, by default, are cells which contain `None`. This can be changed with `options.null_value`. Null cells can be set with `zd` (set current cell to `None`) or `gzd` (set selected rows in current column to `None`). Null cells are distinguished with a yellow ∅' symbol on the right hand corner. They are distinct from empty cells (which are `''` in columns of string type.)
+
+1. Type `|` followed by *.* to select all rows with empty and null cells in the current column.
+
+---
+
+## How to select rows where the current column is null?
+
+There are several different options:
+
+- Move to an empty or null cell in the column of interest and press `,` to select all empty cells in that column.
+- Open a **DescribeSheet** for the current sheet with `Shift+I`. Move to the **nulls** column, and then move to the row which references the source column of interest. Type `zd` to select all null rows for that column.
+- For non-numerical columns `z|` followed by **not ColumnName**, will select all empty cells for that column. For numerical columns it will also select cells with `0`.
+---
+
 ## How to move, copy and remove rows
 
 Command(s)         Operation

plugins/plugins.jsonl

@@ -10,4 +10,4 @@
 {"name": "genericSQL", "description": "add loaders for SQL databases (Oracle, MySQL)", "maintainer": "Andrew Swanson @aswan89", "latest_release": "2020-04-15", "url": "https://raw.githubusercontent.com/aswan89/visidata_plugin_genericSQL/1.0.0/generic_sql.py", "latest_ver": "1.0.0", "visidata_ver": "2.-3", "pydeps": "sqlalchemy cx_oracle mysqlclient pyodbc", "vdplugindeps": "", "sha256": "71bbaabaa4ffb973ef7745dc3eadca108b34a7e0f0bf6cf6631eca3fa4532513"}
 {"name": "diff", "description": "adds command to create diff sheets", "maintainer": "Anja Kefala @anjakefala", "latest_release": "2020-10-09", "url": "https://raw.githubusercontent.com/saulpw/visidata/492d558ff4aba55a192965a27ae6499f8d219072/plugins/diff.py", "latest_ver": "0.9", "visidata_ver": "2.0", "sha256": "0085bf8afdc9abf74b7dd52c251f7b7f7befc507aae262bbb656c2ab7379ddd1"}
 {"name": "marks", "description": "adds commands for marking selected rows, and selecting + viewing marked rows", "maintainer": "Saul Pwanson @saulpw", "latest_release": "2020-10-09", "url": "https://raw.githubusercontent.com/saulpw/visidata/54e49feee97a607cce5355bf0e3a79b2466c3d8a/plugins/marks.py", "latest_ver": "0.1", "visidata_ver": "2.0", "sha256": "f15977f327ccc387be208922b66afe4b65666912e8066d02352c53b1aaff941f"}
-{"name": "conll", "description": "CoNLL data loader", "maintainer": "Paul McCann <polm@dampfkraft.com>", "latest_release": "2020-11-09", "url": "https://raw.githubusercontent.com/polm/visidata-conll/83579a939813b3a3bca638bbb15bb9e8cf4e08ac/conll.py", "latest_ver": "0.1.0", "visidata_ver": "2.0", "pydeps": "pyconll", "sha256": "ff0df4c817121e57780ddd3e16ae67703fa69129c9e33e83fe40dd5220ad93e2"}
+{"name": "conll", "description": "CoNLL data loader", "maintainer": "Paul McCann <polm@dampfkraft.com>", "latest_release": "2020-12-07", "url": "https://raw.githubusercontent.com/polm/visidata-conll/1bc43a825b329c833ff3f9eb8377d850d58ec4cd/conll.py", "latest_ver": "v0.1.1", "visidata_ver": "v2.1", "pydeps": "pyconll", "sha256": "7a89537e1ad173d9cf8ec934cb81836be2e4ddcdc06c7632af92e6754594285c"}

setup.py

@@ -3,7 +3,7 @@
 from setuptools import setup
 # tox can't actually run python3 setup.py: https://github.com/tox-dev/tox/issues/96
 #from visidata import __version__
-__version__ = '2.1'
+__version__ = '2.1.1'
 
 setup(name='visidata',
       version=__version__,

tests/curcol.vd

@@ -0,0 +1,6 @@
+sheet	col	row	longname	input	keystrokes	comment
+			open-file	sample_data/sample.tsv	o	
+sample	Region		addcol-expr	curcol.casefold()	=	create new column from Python expression, with column names as variables
+sample	Item		addcol-expr	curcol.casefold()	=	create new column from Python expression, with column names as variables
+sample	curcol.casefold()		rename-col	Region_col	^	edit name of current column
+sample	curcol.casefold()		rename-col	Item_col	^	edit name of current column

tests/golden/curcol.tsv

@@ -0,0 +1,44 @@
+OrderDate	Region	Region_col	Rep	Item	Item_col	Units	Unit_Cost	Total
+2016-01-06	East	east	Jones	Pencil	pencil	95	1.99	189.05
+2016-01-23	Central	central	Kivell	Binder	binder	50	19.99	999.50
+2016-02-09	Central	central	Jardine	Pencil	pencil	36	4.99	179.64
+2016-02-26	Central	central	Gill	Pen	pen	27	19.99	539.73
+2016-03-15	West	west	Sorvino	Pencil	pencil	56	2.99	167.44
+2016-04-01	East	east	Jones	Binder	binder	60	4.99	299.40
+2016-04-18	Central	central	Andrews	Pencil	pencil	75	1.99	149.25
+2016-05-05	Central	central	Jardine	Pencil	pencil	90	4.99	449.10
+2016-05-22	West	west	Thompson	Pencil	pencil	32	1.99	63.68
+2016-06-08	East	east	Jones	Binder	binder	60	8.99	539.40
+2016-06-25	Central	central	Morgan	Pencil	pencil	90	4.99	449.10
+2016-07-12	East	east	Howard	Binder	binder	29	1.99	57.71
+2016-07-29	East	east	Parent	Binder	binder	81	19.99	1619.19
+2016-08-15	East	east	Jones	Pencil	pencil	35	4.99	174.65
+2016-09-01	Central	central	Smith	Desk	desk	2	125.00	250.00
+2016-09-18	East	east	Jones	Pen Set	pen set	16	15.99	255.84
+2016-10-05	Central	central	Morgan	Binder	binder	28	8.99	251.72
+2016-10-22	East	east	Jones	Pen	pen	64	8.99	575.36
+2016-11-08	East	east	Parent	Pen	pen	15	19.99	299.85
+2016-11-25	Central	central	Kivell	Pen Set	pen set	96	4.99	479.04
+2016-12-12	Central	central	Smith	Pencil	pencil	67	1.29	86.43
+2016-12-29	East	east	Parent	Pen Set	pen set	74	15.99	1183.26
+2017-01-15	Central	central	Gill	Binder	binder	46	8.99	413.54
+2017-02-01	Central	central	Smith	Binder	binder	87	15.00	1305.00
+2017-02-18	East	east	Jones	Binder	binder	4	4.99	19.96
+2017-03-07	West	west	Sorvino	Binder	binder	7	19.99	139.93
+2017-03-24	Central	central	Jardine	Pen Set	pen set	50	4.99	249.50
+2017-04-10	Central	central	Andrews	Pencil	pencil	66	1.99	131.34
+2017-04-27	East	east	Howard	Pen	pen	96	4.99	479.04
+2017-05-14	Central	central	Gill	Pencil	pencil	53	1.29	68.37
+2017-05-31	Central	central	Gill	Binder	binder	80	8.99	719.20
+2017-06-17	Central	central	Kivell	Desk	desk	5	125.00	625.00
+2017-07-04	East	east	Jones	Pen Set	pen set	62	4.99	309.38
+2017-07-21	Central	central	Morgan	Pen Set	pen set	55	12.49	686.95
+2017-08-07	Central	central	Kivell	Pen Set	pen set	42	23.95	1005.90
+2017-08-24	West	west	Sorvino	Desk	desk	3	275.00	825.00
+2017-09-10	Central	central	Gill	Pencil	pencil	7	1.29	9.03
+2017-09-27	West	west	Sorvino	Pen	pen	76	1.99	151.24
+2017-10-14	West	west	Thompson	Binder	binder	57	19.99	1139.43
+2017-10-31	Central	central	Andrews	Pencil	pencil	14	1.29	18.06
+2017-11-17	Central	central	Jardine	Binder	binder	11	4.99	54.89
+2017-12-04	Central	central	Jardine	Binder	binder	94	19.99	1879.06
+2017-12-21	Central	central	Andrews	Binder	binder	28	4.99	139.72

tests/golden/listofdictobj.tsv

@@ -0,0 +1,3 @@
+imag	real
+0.00	6.08
+0.00	50.77

tests/listofdictobj.vd

@@ -0,0 +1,4 @@
+sheet	col	row	longname	input	keystrokes	comment
+			open-file	sample_data/y77d-th95.json.gz	o	
+y77d-th95	geolocation		expand-col-depth	1	z(	expand current column of containers to given depth (0=fully)
+	geolocation.coordinates	0	open-cell		z^J	open sheet with copies of rows referenced in current cell

visidata/__init__.py

@@ -1,6 +1,6 @@
 'VisiData: a curses interface for exploring and arranging tabular data'
 
-__version__ = '2.1'
+__version__ = '2.1.1'
 __version_info__ = 'VisiData v' + __version__
 __author__ = 'Saul Pwanson <vd@saul.pw>'
 __status__ = 'Production/Stable'

visidata/_open.py

@@ -42,7 +42,10 @@ def openPath(vd, p, filetype=None):
     'Call ``open_<filetype>(p)`` or ``openurl_<p.scheme>(p, filetype)``.  Return constructed but unloaded sheet of appropriate type.'
     if p.scheme and not p.fp: # isinstance(p, UrlPath):
         openfunc = 'openurl_' + p.scheme
-        return vd.getGlobals()[openfunc](p, filetype=filetype)
+        try:
+            return vd.getGlobals()[openfunc](p, filetype=filetype)
+        except KeyError:
+            vd.fail(f'no loader for url scheme: {p.scheme}')
 
     if not filetype:
         if p.is_dir():

visidata/basesheet.py

@@ -226,6 +226,11 @@ def sheet(self):
     'the top sheet on the stack'
     return self.sheets[0] if self.sheets else None
 
+@VisiData.api
+def isLongname(self, ks):
+    'Return True if *ks* is a longname.'
+    return ('-' in ks) and (ks[-1] != '-') or (len(ks) > 3 and ks.islower())
+
 
 @VisiData.api
 def getSheet(vd, sheetname):

visidata/cmdlog.py

@@ -147,7 +147,7 @@ def beforeExecHook(self, sheet, cmd, args, keystrokes):
             self.afterExecSheet(sheet, False, '')
 
         colname, rowname, sheetname = '', '', None
-        if sheet and not (cmd.longname.startswith('open-') and cmd.longname != 'open-row'):
+        if sheet and not (cmd.longname.startswith('open-') and not cmd.longname in ('open-row', 'open-cell')):
             sheetname = sheet
 
             contains = lambda s, *substrs: any((a in s) for a in substrs)
@@ -281,12 +281,13 @@ def activeSheet(vd):
 def replayOne(vd, r):
         'Replay the command in one given row.'
         vd.currentReplayRow = r
-        if r.sheet:
+        longname = getattr(r, 'longname', None)
+
+        if r.sheet and not (r.sheet == 'override' and longname in ['set-option', 'unset-option']):
             vs = vd.getSheet(r.sheet) or vd.error('no sheet named %s' % r.sheet)
         else:
             vs = None
 
-        longname = getattr(r, 'longname', None)
         if longname in ['set-option', 'unset-option']:
             try:
                 context = vs if r.sheet and vs else vd

visidata/help.py

@@ -36,7 +36,7 @@ def iterload(self):
         self.revbinds = {}  # [longname] -> keystrokes
         itbindings = vd.bindkeys.iterall()
         for (keystrokes, _), longname in itbindings:
-            if (keystrokes not in self.revbinds) and ('-' not in keystrokes or keystrokes[-1] == '-'):
+            if (keystrokes not in self.revbinds) and not vd.isLongname(keystrokes):
                 self.revbinds[longname] = keystrokes
 
 

visidata/loaders/postgres.py

@@ -112,7 +112,11 @@ def openRow(self, row):
 class PgTable(Sheet):
     @asyncthread
     def reload(self):
-        with self.sql.cur("SELECT * FROM " + self.source) as cur:
+        if self.options.postgres_schema:
+            source = f"{self.options.postgres_schema}.{self.source}"
+        else:
+            source = self.source
+        with self.sql.cur(f"SELECT * FROM {source}") as cur:
             self.rows = []
             r = cur.fetchone()
             if r:

visidata/macros.py

@@ -24,7 +24,10 @@ def runMacro(vd, macro):
 
 def setMacro(ks, vs):
     vd.macrobindings[ks] = vs
-    BaseSheet.addCommand(ks, vs.name, 'runMacro(vd.macrobindings[keystrokes])')
+    if vd.isLongname(ks):
+        BaseSheet.addCommand('', ks, 'runMacro(vd.macrobindings[longname])')
+    else:
+        BaseSheet.addCommand(ks, vs.name, 'runMacro(vd.macrobindings[keystrokes])')
 
 
 @CommandLog.api

visidata/main.py

@@ -2,7 +2,7 @@
 # Usage: $0 [<options>] [<input> ...]
 #        $0 [<options>] --play <cmdlog> [--batch] [-w <waitsecs>] [-o <output>] [field=value ...]
 
-__version__ = '2.1'
+__version__ = '2.1.1'
 __version_info__ = 'saul.pw/VisiData v' + __version__
 
 from copy import copy
@@ -297,6 +297,8 @@ def vd_cli():
     rc = -1
     try:
         rc = main_vd()
+    except BrokenPipeError:
+        os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno()) # handle broken pipe gracefully
     except visidata.ExpectedException as e:
         print('Error: ' + str(e))
     except FileNotFoundError as e:

visidata/man/vd.1

@@ -1198,7 +1198,7 @@ Core VisiData includes these sources:
 .It Sy sqlite
 .Bl -inset -compact -offset xxx
 .It May include multiple tables. The initial sheet is the table directory;
-.Sy Enter No loaders the entrie table into memory. Sy z^S No saves modifications to source.
+.Sy Enter No loads the entire table into memory. Sy z^S No saves modifications to source.
 .El
 .El
 .Pp

visidata/man/vd.inc

@@ -890,7 +890,7 @@ Core VisiData includes these sources:
 .It Sy sqlite
 .Bl -inset -compact -offset xxx
 .It May include multiple tables. The initial sheet is the table directory;
-.Sy Enter No loaders the entrie table into memory. Sy z^S No saves modifications to source.
+.Sy Enter No loads the entire table into memory. Sy z^S No saves modifications to source.
 .El
 .El
 .Pp

visidata/man/vd.txt

@@ -868,7 +868,7 @@ SUPPORTED SOURCES
 
         sqlite
            May include multiple tables. The initial sheet is the table
-           directory; Enter loaders the entrie table into memory. z^S saves
+           directory; Enter loads the entire table into memory. z^S saves
            modifications to source.
 
      URL schemes are also supported:

visidata/pyobj.py

@@ -152,10 +152,10 @@ def SheetList(*names, **kwargs):
         vd.status('no content in %s' % names)
         return
 
-    if isinstance(src, dict):
+    if isinstance(src[0], dict):
         return ListOfDictSheet(*names, **kwargs)
-    elif isinstance(src, tuple):
-        if getattr(src, '_fields', None):  # looks like a namedtuple
+    elif isinstance(src[0], tuple):
+        if getattr(src[0], '_fields', None):  # looks like a namedtuple
             return ListOfNamedTupleSheet(*names, **kwargs)
 
     # simple list

visidata/save.py

@@ -68,6 +68,8 @@ def itervals(sheet, *cols, format=False):
 @Sheet.api
 def getDefaultSaveName(sheet):
     src = getattr(sheet, 'source', None)
+    if hasattr(src, 'scheme') and src.scheme:
+        return src.name + src.suffix
     if isinstance(src, Path):
         return str(src)
     else: