Merge pull request #883 from saulpw/develop

v2.2

CHANGELOG.md

@@ -1,5 +1,71 @@
 # VisiData version history
 
+# v2.2 (2021-01-30)
+
+## Options
+
+    - [cli options] now global by default; use `-n` to set option as sheet-specific instead
+        - add `-n`/`--nonglobal` to make subsequent CLI options "sheet-specific" (applying only to paths specified directly on the CLI)
+        - keep `-g`/`--global` to make subsequent CLI options "global" (applying to all sheets by default unless overriden)
+        - invert the default: now CLI options are global by default (thus `-g` is a no-op unless preceded by `-n` on the CLI)
+        - `-g` no longer acts as a toggle
+
+    - [input] add `options.input_history` (thanks @tsibley and @ajkerrigan #468)
+        - basename of file to store persistent input history (default of `''` means disabled)
+        - caveat: persistent file only read if option given before first input
+
+    - [options.fancy_chooser] now disabled by default--use `Ctrl+X` to open from a choose() prompt
+
+## Types
+
+    - [types] add `floatlocale` type (thanks @Guiriguanche #863)
+        - add commands `type-floatlocale` and `type-floatlocale-selected` (unbound by default)
+        - `floatlocale` parses based on `LC_NUMERIC` envvar (must be set before launching)
+        - parsing is 20x slower than with standard float column
+        - will parse commas as decimals (e.g. '1,1') if LC_NUMERIC is set to a locale like 'en_DK.UTF-8'
+
+## Loaders
+
+    - [loaders geojson] add loading and saving support for geojson files (thanks @draco #876)
+    - [loaders vds] add loader/saver for custom .vds format (VisiData Sheet) to save column properties and data for multiple sheets in one file
+    - [ux] autoload all subsheets by default; set `options.load_lazy` to disable
+        - removes a minor friction with unloaded subsheets
+
+    - [loaders http] add `options.http_max_next` to limit api pagination (default 0 - no pagination) (thanks @aborruso #830)
+
+## Bugfixes and Adjustments
+
+    - [cli] fail properly if path cannot be opened
+    - [defer] only mention number of deleted rows, if some were deleted
+    - [go-pageup go-pagedown] ensure cursor stays in the same relative positions
+    - [loaders mysql] fix mysql loader duplicating tables for each database (thanks @SuRaMoN #868)
+    - [loaders mysql] perform asynchronous data fetch for mysql loader (thanks @SuRaMoN #869)
+    - [loaders pandas] fix empty subsets for dup-selected and frequency table `open-row` (thanks @ajkerrigan #881 #878)
+    - [loaders shp] fix display (thanks @dracos #874)
+    - [loaders shp] fix saving to geojson (thanks @dracos #876)
+    - [replay] fix replaying of .vd with `set-option`
+    - [slide] fix bug when sliding key columns to the left, after sliding them to the right
+    - [types] add command `type-floatsi-selected` on **Columns Sheet**
+
+    - [expand] errors and nulls can now be expanded with `expand-cols` (thanks @geekscrapy #865)
+
+    - [open] openSource now uses **'global'** `options.filetype` instead of sheet-specific as previous
+        - to set the filetype for a file locally, set through cli: `vd -f tsv sample.foo`
+        - to set in the **CommandLog**, use sheet="global" with longname="set-option"
+
+    - [loaders http] raise exception if http status is not 20x (thanks @geekscrapy #848)
+    - [loaders shp] support more Shapefile types (thanks @dracos #875)
+
+## API
+    - add `create` kwarg to `openSource()`, to create the file if it does not exist already
+    - [settings] 'global' is now 'default', and 'override' is 'global'
+        - 'default' is the default setting within VisiData
+        - 'global' is a user override on that default that applies globally
+        - sheet-specific overrides global and default, for the sheet it is specific to
+        - options set through visidatarc and cli are 'global' unless otherwise specified
+    - [save] grab `save_foo` from **SheetType** first
+        - allows overrides of sheet-specific saving
+
 # v2.1.1 (2021-01-03)
 
     - [macros] allow macro interfaces to be longnames (thanks @frosencrantz #787)

README.md

@@ -1,5 +1,5 @@
 
-# VisiData v2.1.1 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/develop.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/develop) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)
+# VisiData v2.2 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)
 
 A terminal interface for exploring and arranging tabular data.
 

dev/checklists/release.md

@@ -111,21 +111,23 @@ git tag v#.#.#
 git push --tags
 ```
 
-15. Write up the release notes and add it to `www/releases.md`
+15. Write up the release notes and add it to `www/releases.md`. Add it to index.md.
 
 16. Upload new motd
 
-17. Run dev/mkwww.sh to rebuild the website and deploy with dev/deploy_www.sh
+17. Update the website by pushing to master.
 
 18. Comb through issues and close the ones that have been solved, referencing the version number
 
-19. Post github release notes on tinyletter.
+19. Post github release notes on patreon.
 
 20. Update the other distributions.
 
 # conda
 
-1. Fork https://github.com/conda-forge/visidata-feedstock
+Registry: https://github.com/conda-forge/visidata-feedstock/
+
+1. Fork https://github.com/conda-forge/visidata-feedstock. Open the recipe/meta.yaml.
 
 2. Update the VisiData version and sha256.
 
@@ -135,44 +137,52 @@ git push --tags
 
 5. Comment `@conda-forge-admin, please rerender` in the PR.
 
+6. Merge the PR, when everything is green.
+
 
 # Homebrew
 
-1. Update the link in url to the new visidata tar.gz file.
-2. Download the tarfile and obtain its new sha256
-```
-shasum -a 256
-```
-3. Check each dependency and see if it has been updated. If so, update the url and sha256 for the newest version.
-4. Install visidata using `pip3 install visidata --upgrade` and note down all of the new dependencies. 
-5. Obtain their urls and sha256 and add them to the formula
-6. Change their urls from `pypi.python.org` to `files.pythonhosted.org`.
-7. Test the formula with `brew install --build-from-source visidata`. Fix as needed.
-8. Audit the formula with `brew audit --new-formula visidata`
-9. Add and commit the formula.
+Registry: https://github.com/saulpw/homebrew-vd
+
+1. Open the Formula/visidata.rb file. Update the link in url to the new visidata tar.gz file.
+2. Update the sha256 and version in visidata.rb. Update the version in README.md.
+3. For major version ships, check each dependency and see if it has been updated. If so, update the url and sha256 for the newest version.
+4. On a mac, test the formula with `brew install --build-from-source visidata`. Fix as needed.
+5. Audit the formula with `brew audit --new-formula visidata`
+6. Add and commit the formula.
 
 ## Debian
-1. Obtain the visidata tar.gz file from pypi
+1. Download the visidata tar.gz file from pypi
 2. tar -xzmf visidata.tar.gz
 3. cp visidata.ver.tar.gz visidata_ver.orig.tar.gz
-4. cd visidata/
-5. Place there the contents of the debian directory from github.com/saulpw/visidata
+4. cd visidata-ver/
+5. Place there the contents of the debian directory from github.com/saulpw/visidata/dev/debian/
 6. Update changelog
 ```
-dch -v version-revision
+dch -v new_version
 ```
-7. Run debuild. Fix errors as they come up
+
+where new_version = "$version"-1
+
+Edit as necessary. I usually set stability to unstable, and urgency to low, and add a small changelog.
+7. Run debuild. Fix errors as they come up.
 8. If a package fails to import a module, it must be added to the build dependencies as python3-modules
 9. cd ..
-10. Sign the changes files
+10. If unsigned by debuild, sign the changes files
 ```
 debsign -k keycode visidata_ver.changes
 ```
-11. Upload to debian mentors
+
+anjakefala has the key and password.
+11. Upload to debian mentors and contact the mentor, [Martin](https://qa.debian.org/developer.php?email=debacle%40debian.org).
 ```
 dput mentors visidata_ver.changes
 ```
 
+12. Copy the fresh contents of the debian folder back into visidata/dev/debian.
+
 ## deb-vd
-9. Enter saulpw/deb-vd.
-10. Run the command reprepro includedeb sid new-vd.deb
+Private registry: https://github.com/saulpw/deb-vd
+1. Enter saulpw/deb-vd.
+2. Run the command reprepro includedeb sid new-vd.deb
+3. Update the README.md with the new version, commit *all* the changes and push to master.

dev/debian/changelog

@@ -1,3 +1,9 @@
+visidata (2.1.1-1) unstable; urgency=low
+
+  * Update package to 2.1.1-1
+
+ -- Anja Boskovic <anja.kefala@gmail.com>  Mon, 04 Jan 2021 20:16:44 -0800
+
 visidata (1.5-1) unstable; urgency=low
 
   * Update package to 1.5-1

dev/formats.jsonl

@@ -8,7 +8,8 @@
 {"filetype": "xls", "aliases": "", "requirements": "xlrd", "format": "Excel spreadsheets", "VisiData loader": "yes", "VisiData saver": "", "version_added": "0.42", "created": "1987", "creator": "Microsoft", "description": "", "open format": "no", "nestable": "", "plottable": "", "format_url": "https://msdn.microsoft.com/en-us/library/office/cc313154(v=office.12).aspx"}
 {"filetype": "fixed", "filetype_url": "#fixed", "aliases": "", "requirements": "", "format": "fixed width text", "VisiData loader": "yes", "VisiData saver": "displayed text (as of 2.1)", "version_added": "0.97", "created": "", "creator": "", "description": "not recommended for new data", "open format": "prehistory", "nestable": "", "plottable": "", "format_url": "https://stackoverflow.com/questions/7666780/why-are-fixed-width-file-formats-still-in-use"}
 {"filetype": "postgres", "filetype_url": "#postgres", "aliases": "", "requirements": "", "format": "PostgreSQL database", "VisiData loader": "yes", "VisiData saver": "", "version_added": "0.97", "created": "1996", "creator": "", "description": "", "open format": "", "nestable": "", "plottable": "", "format_url": ""}
-{"filetype": "vd", "filetype_url": "#vd", "aliases": "vdj", "requirements": "", "format": "VisiData command log", "VisiData loader": "yes", "VisiData saver": "", "version_added": "0.97", "created": "2017", "creator": "VisiData", "description": "replayable", "open format": "yes", "nestable": "", "plottable": "", "format_url": "http://visidata.org/docs/save-restore/"}
+{"filetype": "vd", "filetype_url": "#vd", "aliases": "vdj", "requirements": "", "format": "VisiData command log", "VisiData loader": "yes", "VisiData saver": "yes", "version_added": "0.97", "created": "2017", "creator": "VisiData", "description": "replayable", "open format": "yes", "nestable": "", "plottable": "", "format_url": "http://visidata.org/docs/save-restore/"}
+{"filetype": "vds", "format": "VisiData Sheet", "VisiData loader": "yes", "VisiData saver": "yes", "version_added": "2.2", "created": "2021", "creator": "VisiData", "description": "save column properties and data for multiple sheets in one file", "open format": "yes"}
 {"filetype": "mbtiles", "filetype_url": "#mbtiles", "aliases": "", "requirements": "mapbox-vector-tile", "format": "MapBox Tileset", "VisiData loader": "yes", "VisiData saver": "", "version_added": "0.98", "created": "2011", "creator": "MapBox", "description": "based on sqlite", "open format": "yes", "nestable": "", "plottable": "plottable", "format_url": "https://docs.mapbox.com/help/glossary/mbtiles/"}
 {"filetype": "pbf", "aliases": "", "requirements": "", "format": "Protocolbuffer Binary Format", "VisiData loader": "yes", "VisiData saver": "", "version_added": "0.98", "created": "2011", "creator": "OpenStreetMap", "description": "OpenStreetMap data file", "open format": "yes", "nestable": "", "plottable": "", "format_url": "https://wiki.openstreetmap.org/wiki/PBF_Format"}
 {"filetype": "shp", "filetype_url": "#shp", "aliases": "dbf", "requirements": "pyshp", "format": "Shapefile geographic data", "VisiData loader": "yes", "VisiData saver": "", "version_added": "0.98", "created": "1993", "creator": "ESRI", "description": "open format created by ESRI", "open format": "yes", "nestable": "", "plottable": "plottable", "format_url": "https://en.wikipedia.org/wiki/Shapefile"}
@@ -18,7 +19,7 @@
 {"filetype": "ttf", "filetype_url": "#ttf", "aliases": "otf", "requirements": "fonttools", "format": "TrueType Font", "VisiData loader": "yes", "VisiData saver": "", "version_added": "1.1", "created": "1991", "creator": "Apple", "description": "", "open format": "yes", "nestable": "", "plottable": "plottable", "format_url": "https://en.wikipedia.org/wiki/TrueType"}
 {"filetype": "dot", "filetype_url": "#pcap", "aliases": "", "requirements": "", "format": "Graphviz diagram", "VisiData loader": " ", "VisiData saver": "from pcap", "version_added": "1.2", "created": "1991", "creator": "", "description": "output for pcap only", "open format": "yes", "nestable": "", "plottable": "", "format_url": "https://graphviz.gitlab.io/_pages/doc/info/lang.html"}
 {"filetype": "dta", "aliases": "", "requirements": "pandas", "format": "Stata", "VisiData loader": "yes", "VisiData saver": "", "version_added": "1.2", "created": "1985", "creator": "StataCorp", "description": "", "open format": "no", "nestable": "", "plottable": "", "format_url": "https://www.loc.gov/preservation/digital/formats/fdd/fdd000471.shtml"}
-{"filetype": "geojson", "filetype_url": "#shp", "aliases": "", "requirements": "", "format": "Geographic JSON", "VisiData loader": " ", "VisiData saver": "from shp", "version_added": "1.2", "created": "2008", "creator": "", "description": "derivative of JSON", "open format": "yes", "nestable": "", "plottable": "plottable", "format_url": "http://geojson.org/"}
+{"filetype": "geojson", "filetype_url": "#shp", "aliases": "", "requirements": "", "format": "Geographic JSON", "VisiData loader": "yes", "VisiData saver": "yes (from shp and geojson)", "version_added": "2.2", "created": "2008", "creator": "", "description": "derivative of JSON", "open format": "yes", "nestable": "", "plottable": "plottable", "format_url": "http://geojson.org/"}
 {"filetype": "sas7bdat", "aliases": "", "requirements": "sas7bdat", "format": "Statistical Analysis System (SAS)", "VisiData loader": "yes", "VisiData saver": "", "version_added": "1.2", "created": "1976", "creator": "SAS Institute", "description": "", "open format": "no", "nestable": "", "plottable": "", "format_url": "https://stackoverflow.com/questions/26532606/how-to-convert-sas7bdat-file-to-csv"}
 {"filetype": "sav", "aliases": "", "requirements": "", "format": "SPSS statistics", "VisiData loader": "yes", "VisiData saver": "", "version_added": "1.2", "created": "1968", "creator": "SPSS Inc", "description": "", "open format": "IBM", "nestable": "", "plottable": "", "format_url": "https://www.dummies.com/education/math/statistics/the-spss-file-format/"}
 {"filetype": "spss", "aliases": "", "requirements": "savReaderWriter", "format": "SPSS statistics", "VisiData loader": "yes", "VisiData saver": "", "version_added": "1.2", "created": "1968", "creator": "SPSS Inc", "description": "", "open format": "IBM", "nestable": "", "plottable": "", "format_url": "https://www.dummies.com/education/math/statistics/the-spss-file-format/"}

dev/history.jsonl

@@ -8,3 +8,4 @@
 {"date": "2020-12-06", "event": "[HN] VisiData in 60 seconds", "url": "https://news.ycombinator.com/item?id=25322091"}
 {"date": "2020-12-05", "event": "[reddit] VisiData - a cool vim-like tool for examining and manipulating data", "url": "https://www.reddit.com/r/vim/comments/kf4wgb/visidata_a_cool_vimlike_tool_for_examining_and/"}
 {"date": "2020-12-06", "event": "RealPython tweets about VisiData", "url": "https://twitter.com/realpython/status/1339669258448547842"}
+{"date": "2021-01-11", "event": "Updated 1.5.2 motd to urge upgrading"}

docs/api/options.rst

@@ -36,7 +36,7 @@ The options context should be referenced directly when setting:
 
     - ``sheet.options`` to *set* an option on a specific sheet instance (**sheet override**).
     - ``<SheetType>.class_options`` to *set* a option default for a particular type of Sheet (**class override**).
-    - ``vd.options`` (or plain ``options``) to *set* a global option default (**global override**).
+    - ``vd.options`` (or plain ``options``) to *set* an option globally for all sheets, except for sheets that have a sheet override (**global override**).
 
 Use ``sheet.options`` to *get* an option within the context of a specific sheet.
 This is strongly preferred, so the user can override the option setting on a sheet-specific basis.

docs/display.md

@@ -14,7 +14,7 @@ Computed properties:
 
 - `cursorRow`: rows[cursorRowIndex]
 - `cursorCol`: columns[cursorColIndex] (equivalent to visibleCols[cursorVisibleColIndex])
-- `nVisibleRows`: simply based on terminal height, the exact number of rows that can be displayed.
+- `nScreenRows`: simply based on terminal height, the exact number of rows that can be displayed.
 - `visibleRows`: slice of onscreen rows; 'visible' has different meaning than visibleCols
 - `visibleCols`: (O(ncols), cached between frames): list of all non-hidden columns
 - `leftVisibleColIndex` is the first non-key column on the left.

docs/plugins.md

@@ -11,7 +11,7 @@ Plugins are optional Python modules that extend or modify base VisiData's functi
 * [jsvine's repo](https://github.com/jsvine/visidata-plugins)
 * [anjakefala's repo](https://github.com/anjakefala/vd-plugins)
 * [ajkerrigan's repo](https://github.com/ajkerrigan/visidata-plugins)
-* ...and [let us know](https://github.com/saulpw/visidata/issues/new) about yours! Advice for [making plugins](https://github.com/saulpw/visidata/blob/develop/dev/checklists/add-plugin.md).
+* ...and [let us know](https://github.com/saulpw/visidata/issues/new) about yours! Advice for [making plugins](https://visidata.org/docs/api).
 
 # How to use/activate a plugin
 
@@ -34,7 +34,7 @@ To install it
 
 ## From within VisiData
 
-We maintain a list of plugins which can be downloaded and installed from within the application itself. To incorporate a plugin into this list, add it to [plugins.jsonl](https://github.com/saulpw/visidata/blob/develop/plugins/plugins.jsonl), and create a PR off of the `develop` branch.  Advice for [making plugins and adding them to our curated sets](https://github.com/saulpw/visidata/blob/develop/dev/checklists/add-plugin.md).
+We maintain a list of plugins which can be downloaded and installed from within the application itself. To incorporate a plugin into this list, add it to [plugins.jsonl](https://github.com/saulpw/visidata/blob/develop/plugins/plugins.jsonl), and create a PR off of the `develop` branch.
 
 To install a plugin, and its dependencies, from within VisiData: