v3.1 Release

.github/ISSUE_TEMPLATE/bug_report.md

@@ -9,17 +9,30 @@ assignees: ''
 
 **Small description**
 
+**Data to reproduce**
+
+[Please include a small amount of sample data that demonstrates the issue.]
+
+**Steps to reproduce**
+
+[If the issue takes more than a few commands, please attach a [saved commandlog](http://visidata.org/docs/save-restore/).]
+
 **Expected result**
 
 **Actual result with screenshot**
-If you get an unexpected error, please include the full stack trace that you get with `Ctrl-E`.
 
-**Steps to reproduce with sample data and a .vd**
-First try reproducing without any user configuration by using the flag `-N`.
-e.g. `echo "abc" | vd -f txt -N`
+[If there is an error message, please include the full stack trace shown with `Ctrl+E`.]
+
+**Configuration**
 
-Please attach the commandlog (saved with `Ctrl-D`) to show the steps that led to the issue.
-See [here](http://visidata.org/docs/save-restore/) for more details.
+- Does this issue reproduce without any plugins or configuration (using the `-N` CLI flag)?
+
+- Does this issue reproduce with either the [latest release](https://www.visidata.org/releases/), or with the [develop branch](https://www.visidata.org/install/#update-visidata-from-an-existing-installation)?
 
 **Additional context**
-Please include the version of VisiData and Python.
+
+- What platform and version are you using (Linux, MacOS, Windows)?
+
+- Which version of Python?
+
+- Which terminal are you using (for display and input issues)?

.github/workflows/main.yml

@@ -14,7 +14,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: [3.7, 3.8, 3.9, "3.10", "3.11"]
+        python-version: [3.8, 3.9, "3.10", "3.11", "3.12"]
 
     runs-on: ubuntu-latest
     timeout-minutes: 20
@@ -43,9 +43,12 @@ jobs:
     - name: Install lxml dependencies
       run: sudo apt install -y libxml2-dev libxslt-dev
 
+    - name: Install ping dependencies
+      run: sudo apt install -y traceroute
+
     - name: Install optional dependencies
       run: |
-        pip3 install -r requirements.txt
+        pip3 install ".[test]"
         pip3 install pytest
 
     - name: Run unit tests
@@ -59,15 +62,17 @@ jobs:
     - name: Test macros
       run: |
         mkdir -p ~/.visidata
-        cp tests/macros/macro* ~/.visidata/
+        cp tests/macros/macros.jsonl tests/macros/1.vdj ~/.visidata/
         vd --batch -p tests/macros/test_macro.vd --output tests/macros/golden/test_macro.tsv
         git --no-pager diff --numstat tests/
         git --no-pager diff --exit-code tests/
 
-    - name: Check docs are built without errors
-      run: |
-        sudo apt install aha
-        dev/mkman.sh
-
     - name: Ensure VisiData can create completions
       run: python -v dev/zsh-completion.py
+
+    - name: Ensure VisiData does not drop lines from RepeatFiles
+      run: |
+        NUMLINES=$(seq 10000 | vd -b | wc -l)
+        if ! [[ $NUMLINES -eq 10000 ]]; then
+            exit 1
+        fi

CHANGELOG.md

@@ -1,5 +1,132 @@
 # VisiData version history
 
+# v3.1 (2024-10-XX)
+
+- drop support for Python 3.7  #2231
+- [vdsql] bump ibis dep to v8
+- [test] fix minimum pandas version to 1.5.3
+
+## Improvements
+
+- [sidebar] Ctrl+G to cycle sidebar (including off) in both main and input modes #2202
+- [sidebar help] rightmost statusline ({vd.sidebarStatus}) shows number of help sidebars (clickable to toggle sidebar)
+- [sheet tabs] left statusline sheet tabs (clickable to jump to sheet) #2030
+- [aggregators] immediate async summary of column aggregators on bottom rows #2209
+- [graph] reflines at user-defined x or y #2487
+- [guides] add several guides (thanks to @thejud @reagle @ajkerrigan @anjakefala)
+- [expressions] much cleanup
+  - `row["colname"]` to refer to a specific column by name, without needing to be a valid identifier #2539
+  - `_row` is underlying row object
+  - support curcol in g= and select and search expressions #2524
+  - $curcol for addcol-sh
+  - globals include most symbols from math by default; also datetime and date both as the vd.date subclass
+  - [input] memory variables included as autocomplete keys #2509
+
+## Tweaks
+
+- [window] pad windows with None so all are same size #2279
+- [history] create visidata_dir (default ~/.visidata/) if not exists to enable input history by default #2298
+- [cli] "-p -" replays stdin as a .vdj file
+- [guide] allow front matter in guide .md files; "sheettype" metadata to associate with a sheet
+- [cmdpalette] do not fuzzymatch keys that start with _
+- [interface] hidden columns minutely visible past right of sheet; `_` to re-expand #2394
+- [interface] record all commands in macro (even replay=False); add status indicator when recording #2435
+- [interface] clickable cell error symbol
+- [interface] default theme color improvements, including a slight bg glow for cursor column
+- [aggregators] add keymin; handle corner cases #2308
+- many small interface improvements: error messages, clarifications, edge cases, guardrails.  (thanks to @midichef for many of these)
+
+## Options
+
+- max_rows to stop loading early #2356
+- [xlsx] add xlsx_color_cells (default: True); set to False to disable xlsx cell colorizers
+- regex_skip is now replayable
+- disp_multiline_focus to only expand cursor row when multiline enabled #2205
+- disp_expert (int); set to 1 to enable advanced interface elements and unhide advanced columns by default; set to 5 (or greater) to disable input palette and help sidebars
+- pass disp_wrap_* to textwrap for multiline display #2506
+- rename note_* to disp_note_* #2381
+- separate color_longname into color_longname_status and color_longname_guide
+
+## Commands
+
+- addcol-histogram to make histogram from any numeric column #2208
+- exec-longname-simple for single-line (non-palette) input
+- go-col-name to go to a column by name
+- [dir] open-row-filetype to open row using the given filetype
+- [graph] plot-source-selected on ColumnsSheet #2424
+- [experimental] sort-selected-asc/desc #2295
+- deprecate view-cell #2381
+- unfurl finishes with column cursor on unfurled column
+
+## API
+
+- [command] addCommand(..., replay=False) to not add to cmdlog
+- [sort] add Sheet.ordering property as list[tuple[Column,bool]] #2295
+- [path] note nonstandard behavior of RepeatFile.read()
+- [input] allow different record kwarg for different fields in inputMultiple
+
+## Loaders
+
+- [msgpack] new loader #2419
+- [grep] new loader for output of grep/ripgrep #2443
+
+- [csv] display loading/saving progress
+- [csv] handle more dialect parameters from Sniffer
+- [csv tsv] remove default regex_skip of hash lines #2458
+- [eml] save email parts without filenames #2376
+- [fixed] skip separating space in fixed_width loader #2255
+- [json] remove column type inference #2475 #2131
+- [json] warn if some cols/sheets were not saved #2199
+- [markdown] use col_href as link for col
+- [ods] strip empty cells/rows from end #2440
+- [rec] roundtrippable
+- [sas] allow edits to source sheet
+- [save] derived null in text formats should be empty string #2476
+- [save] handle empty delimiter when options.safety_first
+- [text] do not strip lines of whitespace #2328
+- [tsv] allow empty regex_skip
+- [tsv] allow saving with NUL delimiters
+- [tsv] if field or row delimiter is NUL, disable regex skipping
+- [vds] allow sheets with no rows #2342
+- [xml] save XML as utf-8 instead of utf-8-sig #2520
+
+## Bugfixes
+
+- [command] addcol-sh: ensure arguments are properly quoted  #2415
+- [selection] make toggle thread-safe for FreqTableSheet #1671
+- [undo] prevent undo of command that created sheet #2244
+- [undo] cmdlog always use rowidx, not rowkey #2246
+- [cliptext] fix onclick url #2466
+- [mouse] fix mouse-disable  #2379
+- [sort] do not modify Sheet._ordering #2254
+- [bindings] do not re-prettykeys; call prettykeys for all bindkey() too #2247
+- [terminal] add no-op for ANSI focus-in and focus-in #2247
+- [sort] add _ordering to Sheet class instead of __init__ #2190
+- [transpose] allow to set values #2548
+- [sheet] make 20 tabs clickable #2030
+- [macro] do not record macro inner steps #2531
+- [filetype] use correct case for file extension determination #2263
+- [replay] stop stderr batch progress when interactive #2251
+- [replay] wait for replay to start before stderr progress #2251
+- [replay] fix rare duplication of replay cmds #2392
+- and many more
+
+# v3.0.2 (2024-01-15)
+
+### Fixes and minor improvements
+
+- [cli] add `--guides` to open the guide index
+- [cmdpalette] only first 10 suggestions should have shortcut keys  #2242
+- [draw] use default bg on col hdr sep
+- [input] draw help sidebar on top of sheet/updater  #2241
+- [graph] sort by x and y columns when diving (PR by @midichef #2226)
+- [guides] improve formatting of command and options in sidebar + guides
+- [replay] stop stderr batch progress when entering interactive mode  #2251
+- [shell] clean up DirSheet sidebar
+- [sidebar] grab user's keystroke of sidebar-toggle for helpstring  #2250
+- [sort] a better fix for maintaining sort ordering on sheet copies  #2190 #2254
+- [tests] add "test" extras for installing PyPI packages needed to run tests
+
 # v3.0.1 (2024-01-07)
 
 ### Fixes and minor improvements

CONTRIBUTING.md

@@ -41,7 +41,7 @@ VisiData is designed to be extensible, and most feature requests can be implemen
 If this would require changes to the VisiData core, and a reasonable design is approved, then the issue can stay open until the core changes have been made.
 Otherwise, in the spirit of Marie Kondo, the issue will be closed without prejudice.
 
-Feature requeste with some amount of working Python code are more likely to get attention.
+Feature requests with some amount of working Python code are more likely to get attention.
 Design proposals with concrete use cases are very welcome.
 
 ## Writing a well constructed bug report
@@ -50,28 +50,28 @@ If you encounter any bugs or have any problems, please [create an issue on GitHu
 
 A great bug report will include:
 
-    - a stacktrace, if there is an unexpected error; the most recent full stack traces can be viewed with `Ctrl+E` (then saved with `Ctrl+S`)
-    - a [.vd](http://visidata.org/docs/save-restore/) and sample dataset that reproduces the issue
-    - a .png/.gif (esp. for user interface changes)
+  - a stacktrace, if there is an unexpected error; the most recent full stack traces can be viewed with `Ctrl+E` (then saved with `Ctrl+S`)
+  - a [.vd](http://visidata.org/docs/save-restore/) and sample dataset that reproduces the issue
+  - a .png/.gif (esp. for user interface changes)
 
 Some examples of great bug reports:
 
-    - [#350 by @chocolateboy](https://github.com/saulpw/visidata/issues/350)
-    - [#340 by @Mikee-3000](https://github.com/saulpw/visidata/issues/340)
+  - [#350 by @chocolateboy](https://github.com/saulpw/visidata/issues/350)
+  - [#340 by @Mikee-3000](https://github.com/saulpw/visidata/issues/340)
 
 
 ## Submitting Source Code
 
 Check out the [Plugin Authors Guide](https://visidata.org/docs/api) for an overview of the API.
-Code in `plugins/` or `visidata/loaders/` is welcome, as long as it is useful to someone and safe for everyone.
+Code in `visidata/features/` or `visidata/loaders/` is generally welcome, as long as it is useful to someone and safe for everyone.
 Updates or additions to the core code should be proposed via an [Github Issue](https://github.com/saulpw/visidata/issues/new/choose) before submitting a PR.
 
 VisiData has two main branches:
 
   - [stable](https://github.com/saulpw/visidata/tree/stable) has the last known good version of VisiData (what is in pypi/brew/apt).
   - [develop](https://github.com/saulpw/visidata/tree/develop) has the most up-to-date version of VisiData (which will eventually be merged to stable).
 
-All pull requests should be submitted against `develop`. Submitters will need to e-sign a Copyright Assignment Agreement (CAA) before a pull request will be accepted.
+All pull requests should be submitted against `develop`. Submitters will need to e-sign a [Copyright Assignment Agreement (CAA)](https://visidata.org/caa) before a pull request will be accepted.
 
 # Open Source License and Copyright
 

MANIFEST.in

@@ -12,3 +12,4 @@ include visidata/experimental/noahs_tapestry/*.json
 include visidata/experimental/noahs_tapestry/*.md
 include visidata/experimental/noahs_tapestry/*.ddw
 include visidata/experimental/noahs_tapestry/*.sqlite
+include visidata/guides/*.md

README.md

@@ -1,4 +1,4 @@
-# VisiData v3.0
+# VisiData v3.1
 
 [![Tests](https://github.com/saulpw/visidata/workflows/visidata-ci-build/badge.svg)](https://github.com/saulpw/visidata/actions/workflows/main.yml)
 [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)
@@ -33,9 +33,16 @@ See [visidata.org/install](https://visidata.org/install) for detailed instructio
 
 ### Usage
 
+On Linux and OS/X
+
     $ vd <input>
     $ <command> | vd
 
+On Windows
+
+    $ visidata <input>
+    $ <command> | visidata
+
 Press `Ctrl+Q` to quit at any time.
 
 Hundreds of other commands and options are also available; see the documentation.

bin/viewtsv.py

@@ -3,7 +3,7 @@
 # as plugin: `from viewtsv import open_tsv` in .visidatarc
 # standalone: `viewtsv.py <tsv-files>'
 
-from visidata import Sheet, ColumnItem, asyncthread, options
+from visidata import VisiData, Sheet, ColumnItem, asyncthread, options
 
 @VisiData.api
 def open_tsv(vd, p):
@@ -37,5 +37,5 @@ def reload(self):
 # a minimal main() for standalone apps
 if __name__ == '__main__':
     import sys
-    from visidata import run, Path
-    run(*(open_tsv(Path(fn)) for fn in sys.argv[1:]))
+    from visidata import run, Path, vd
+    run(*(vd.open_tsv(Path(fn)) for fn in sys.argv[1:]))

dev/ROADMAP

@@ -40,11 +40,7 @@
 6. default theme
   - tooling to make it easier to browse color combos
 
-7. website
-  - add vd.org/contributors
-  - add vd.org/pricing
-
-8. new clients
+7. new clients
 - vd tui client: connect to vd server as separate terminal process
 - vd web client
   - lightweight JS widget to display/interact with command server

dev/checklists/feature.md

@@ -0,0 +1,34 @@
+
+# Feature Checklist
+
+- [ ] user docs in docs/
+- [ ] API docs in docs/api/
+- [ ] release notes (in changelog until release)
+- [ ] add to labeled screenshot (if feature visible in main UI)
+- [ ] screenshot image and/or animated gif (if feature visible in UI)
+- [ ] passing pytest (all test_ functions in the module/feature)
+
+## Marketing
+
+- [ ] video demo
+- [ ] blog post
+- [ ] mastodon post
+
+# Loader sample data
+
+- [ ] small amount of sample data in `sample_data`, including (when supported by the format/loader):
+  - [ ] compression
+  - [ ] null value  ("N/A" or "" or None)
+  - [ ] error value ("#ERR")
+  - [ ] int type
+  - [ ] float type
+  - [ ] date type
+  - [ ] nested data
+    - [ ] list of homogenous values
+    - [ ] dict (subrecord)
+  - [ ] multiple sheets
+  - [ ] non-ascii characters
+
+
+- [ ] extra data for special features (e.g. .zip encryption)
+

dev/checklists/manual-tests.md

@@ -94,3 +94,5 @@
 22. Test `open-row` on an html link: https://hls.gsfc.nasa.gov/data/
 23. That DirSheet requires a commit-sheet before changes on filesystem
 24. Test adding multiple aggregators via palette (+)
+25. time vd -p tests/quit-nosave.vdj  - note down the time. compare to PR #2369
+26. Use the z; command. Then type in a command line like echo "| Ceci n'est pas une pipe"

dev/checklists/release.md

@@ -11,10 +11,6 @@
     c. visidata.org; (formats?)
         - remember to check the tables with prettier: https://github.com/saulpw/visidata/pull/2056
 
-3. Verify that VisiData is up-to-date:
-    * help menu
-    * plugins.jsonl
-
 3. Ensure `develop` automated tests run correctly with dev/test.sh
 
 4. Go through the manual tests checklist