diff --git a/CHANGELOG.md b/CHANGELOG.md index f5276ffc5..690763eed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,24 @@ # VisiData version history +# v0.99 (2017-12-22) + +- tab completion for filename and python expr +- `v` now 'visibility toggle' (moved from `w`) +- `^W` to erase a word in the line editor +- `gC` +- `--version` (thanks for the suggestion, @jsvine) +- `options.use_default_colors` +- `median` aggregator +- .html loads tables (requires lxml) + - simple http works (requires requests) +- json save +- json incremental load +- [cmdlog] use rowkey if available instead of row number; options.rowkey_prefix +- [cmdlog] only set row/col when relevant +- [vdtui] task renamed to thread +- /howto/dev/loader +- /design/graphics + # v0.98.1 (2017-12-04) - [packaging] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6e49b7933..1b3718a57 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,11 +17,11 @@ In addition, the innermost core file, `vdtui.py`, is a single-file stand-alone l ### Feature requests or enhancements -If you use VisiData, I would love it if you reached out to me to discuss some of your common workflows and needs. This helps me better prioritize which functionality to add. Send me a [screencast](http://asciinema.org), or some sample data along with your desired output. There is probably a way to tweak VisiData to get the job done even more to your liking. Feature requests should be made on any of the communication channels listed [here](https://github.com/saulpw/visidata/blob/develop/CONTRIBUTING.md#community). +If you use VisiData, I would love it if you reached out to me to discuss some of your common workflows and needs. This helps me better prioritize which functionality to add. Send me a [screencast](http://asciinema.org), or some sample data along with your desired output. There is probably a way to tweak VisiData to get the job done even more to your liking. Feature requests should be made on any of the communication channels listed [here](https://github.com/saulpw/visidata/blob/stable/CONTRIBUTING.md#community). ### Filing issues or support -Create a GitHub issue if anything doesn't appear to be working right. If you get an unexpected error, please include the full stack trace that you get with `^E`. Additionally, it would help me more quickly diagnose the problem if you could attach the saved Commandlog (`Ctrl-D`), which will show the steps that lead to the issue. If you are struggling with learning how to use the tool, or are unsure how something works, please also file an issue or write a comment in any of our [community spaces](https://github.com/saulpw/visidata/blob/develop/CONTRIBUTING.md#community). In addition to wanting to help users get the most out of the tool, this helps us gauge the holes in our documentation. +Create a GitHub issue if anything doesn't appear to be working right. If you get an unexpected error, please include the full stack trace that you get with `Ctrl-E`. Additionally, it would help me more quickly diagnose the problem if you could attach the saved Commandlog (`Ctrl-D`), which will show the steps that lead to the issue. If you are struggling with learning how to use the tool, or are unsure how something works, please also file an issue or write a comment in any of our [community spaces](https://github.com/saulpw/visidata/blob/stable/CONTRIBUTING.md#community). In addition to wanting to help users get the most out of the tool, this helps us gauge the holes in our documentation. ### Contributing tests @@ -33,7 +33,7 @@ To run a test manually: ``` $ bin/vd --play tests/foo.vd --replay-wait 1 - or $ bin/vd -p tests/foo.vd -d 1 + or $ bin/vd -p tests/foo.vd -w 1 ``` To build a `.vd` file: @@ -42,15 +42,17 @@ To build a `.vd` file: 2. Press `Shift-D` to view the commandlog. 3. Edit the commandlog to minimize the number of commands. Cells may be parameterized like `{foo}`, to be specified on the commandline: - $ vd cmdlog.vd --foo=value +``` + $ vd cmdlog.vd --foo=value +``` -4. Press `^S` to save the commandlog to a `.vd` file. +4. Press `Ctrl-S` to save the commandlog to a `.vd` file. -As a shortcut, `^D` will save the current commandlog, by default to the next non-existing 'cmdlog-#.vd' +As a shortcut, `Ctrl-D` will save the current commandlog, by default to the next non-existing 'cmdlog-#.vd' ### Contributing to documentation -If you notice a `globalCommand()` or `Command()` which does not have an entry in the VisiData manpage, please file an issue. In addition, if something is not clear (and in fact, is confusing) let us know so that we can better improve on the documentation. +If you notice a `globalCommand()` or `Command()` which does not have an entry in the vd manpage, please file an issue. In addition, if something is not clear (and in fact, is confusing) let us know so that we can better improve on the documentation. If you would like to contribute by building an asciicast, the process is shown at [visidata.org/test/meta](http://visidata.org/test/meta). diff --git a/README.md b/README.md index 667d1a85e..cb7cfa08b 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# VisiData v0.98.1 [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) +# VisiData v0.99 [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) A terminal interface for exploring and arranging tabular data. @@ -7,12 +7,21 @@ A terminal interface for exploring and arranging tabular data. - Linux or OS/X - Python 3.4+ - python-dateutil -- other [modules may be required](https://github.com/saulpw/visidata/blob/stable/requirements.txt) for opening particular data sources - - for a breakdown, see [supported sources](http://visidata.org/man/) in the VisiData manpage +- other modules may be required for opening particular data sources + - see [requirements.txt](https://github.com/saulpw/visidata/blob/stable/requirements.txt) or the [supported sources](http://visidata.org/man/#loaders) in the vd manpage -## Install +## Installation -To install VisiData, with loaders for formats supported by the Python standard library (which includes csv, tsv, fixed-width text, json, sqlite and graphs): +There are three options for installing visidata: +- [pip3](https://github.com/saulpw/visidata/tree/stable#install-via-pip3) for users who wish to import visidata into their own code or wish to integrate it into a python virtual environment +- [Homebrew](https://github.com/saulpw/visidata/tree/stable#install-via-brew) on MacOS/X for reliable installation of application components (such as the manpage) +- [apt](https://github.com/saulpw/visidata/tree/stable#install-via-apt) on Linux distributions + +### Install via pip3 + +Best installation method for users who wish to take advantage of VisiData in their own code, or integrate it into a Python3 virtual environment. + +To install VisiData, with loaders for the most common data file formats (including csv, tsv, fixed-width text, json, sqlite, http, html and xls): ``` $ pip3 install visidata @@ -24,6 +33,45 @@ To install VisiData, plus external dependencies for all available loaders: pip3 install "visidata[full]" ``` +### Install via brew + +Ideal for MacOS users who primarily want to engage with VisiData as an application. This is currently the most reliable way to install VisiData's manpage on MacOS. + +``` +brew install saulpw/vd/visidata +``` + +Further instructions available [here](https://github.com/saulpw/homebrew-vd). + +### Install via apt + +Packaged for Linux users who do not wish to wrangle with PyPi or python3-pip. + +Currently, VisiData is undergoing review for integration into the main Debian repository. Until then it is available in our [Debian repo](https://github.com/saulpw/deb-vd). + +Grab our public key + +``` +wget http://visidata.org/devotees.gpg.key +apt-key add devotees.gpg.key +``` + +Add our repository to apt's search list + +``` +sudo apt-get install apt-transport-https +sudo vim /etc/apt/sources.list + deb[arch=amd64] https://raw.githubusercontent.com/saulpw/deb-vd/master sid main +sudo apt-get update +``` +You can then install VisiData by typing: + +``` +sudo apt-get install visidata +``` + +Further instructions available [here](https://github.com/saulpw/deb-vd). + ## Run ``` @@ -31,7 +79,7 @@ $ vd [] ... $ | vd [] ``` -VisiData supports tsv, csv, xlsx, hdf5, sqlite, and more. +VisiData supports tsv, csv, xlsx, hdf5, sqlite, json and more. Use `-f ` to force a particular filetype. ## Documentation @@ -50,21 +98,23 @@ For more detailed information about how you can contribute as a developer, influ ## vdtui -The core `vdtui.py` can be used to quickly create efficient terminal workflows. +The core `vdtui.py` can be used to quickly create efficient terminal workflows. These have been prototyped as proof of this concept: - [vgit](https://github.com/saulpw/vgit): a git interface +- [vsh](http://github.com/saulpw/vsh): a collection of utilities like `vping` and `vtop`. - [vdgalcon](https://github.com/saulpw/vdgalcon): a port of the classic game [Galactic Conquest](https://www.galcon.com) -Other workflows should also be created as separate apps using vdtui. These apps can be very small; for example, see the included [viewtsv](bin/viewtsv). +Other workflows should also be created as separate apps using vdtui. These apps can be very small and provide a lot of functionality; for example, see the included [viewtsv](bin/viewtsv). ## License The innermost core file, `vdtui.py`, is a single-file stand-alone library that provides a solid framework for building text user interface apps. It is distributed under the MIT free software license, and freely available for inclusion in other projects. -Other VisiData components, including the main `vd` application, addons, loaders, and other code in this repository, are available for reuse under GPLv3. +Other VisiData components, including the main `vd` application, addons, loaders, and other code in this repository, are available for use and distribution under GPLv3. ## Credits -VisiData was created by Saul Pwanson ``. -Thanks to @anjakefala for test and release support, to @databranner for documentation, and to those wonderful users who contribute feedback in any form, for helping to make VisiData the awesome tool that it is. +VisiData was created and developed by Saul Pwanson ``. + +Thanks to all the [contributors](CONTRIBUTING.md#contributors), and to those wonderful users who provide feedback, for making VisiData the awesome tool that it is. diff --git a/RELEASE.md b/RELEASE.md new file mode 100644 index 000000000..bc15ddcb1 --- /dev/null +++ b/RELEASE.md @@ -0,0 +1,126 @@ +# Release process for the next `stable` version + +1. Merge `stable` to `develop` (if necessary) + +2. Ensure `develop` automated tests run correctly with ./test.sh + +3. Verify that documentation/docstrings are up-to-date on features and functionality + +4. Verify that setup.py is up-to-date with requirements. + +5. Set version number to next most reasonable number (v#.#.#) + + a. add to front of CHANGELOG, along with the release date and bullet points of major changes + + b. update version number on README and front page of website + + c. bump version in `__version__` in source code (bin/vd, visidata/vdtui.py) and setup.py + +6. Run ./mkwww.sh to build the manpage and updated website + +7. Push `develop` to testpypi + + a. set up a ~/.pypirc + + ``` + [distutils] + index-servers= + pypi + testpypi + [pypi] + repository:https://upload.pypi.org/legacy/ + username: + password: + + [testpypi] + repository: https://test.pypi.org/legacy + username: + password: + ``` + + b. push to testpypi + + ``` + python3 setup.py sdist + twine upload dist/* -r testpypi + ``` + +8. Test install from testpypi + + a. on virgin instance + + b. on mac and linux + + c. See if windows works + + d. from git clone + + ``` + pip3 install --extra-index-url https://test.pypi.org/project visidata + ``` + +9. Merge `develop` to stable + +10. Merge `stable` back into other branches + + a. if the branch works with minimal conflicts, keep the branch + + b. otherwise, clean out the branch + + +11. Push stable to pypi + +``` +twine upload dist/* +``` + +12. Test install/upgrade from pypi + + a. Build and deploy the website + + b. Ask someone else to test install + +13. Create a tag `v#.#.#` for that commit +``` +git tag v#.#.# +git push --tags +``` + +14. Push code to stable + +15. Write up the release notes and post at visidata.org/release/#.# + +16. Comb through issues and close the ones that have been solved, referencing the version number + +17. Post release notes on r/visidata and tinyletter and have some ice cream + + +# Homebrew + +1. Update the link in url to the new visidata tar.gz file. +2. Download the tarfile and obtain its new sha256 +``` +shasum -a 256 +``` +3. Check each dependency and see if it has been updated. If so, update the url and sha256 for the newest version. +4. Install visidata using `pip3 install visidata --upgrade` and note down all of the new dependencies. +5. Obtain their urls and sha256 and add them to the formula +6. Change their urls from `pypi.python.org` to `files.pythonhosted.org`. +7. Test the formula with `brew install --build-from-source visidata`. Fix as needed. +8. Audit the formula with `brew audit --new-formula visidata` +9. Add and commit the formula. + +## Debian +1. Obtain the visidata tar.gz file from pypi +2. tar -xzmf visidata.tar.gz +3. cp visidata.ver.tar.gz visidata_ver.orig.tar.gz +4. cd visidata/ +5. Place there the contents of the debian directory from github.com/saulpw/visidata +6. Update changelog +``` +dch -v version-revision +``` +7. Run debuild. Fix errors as they come up +8. If a package fails to import a module, it must be added to the build dependencies as python3-modules +9. Enter saulpw/deb-vd. +10. Run the command reprepro includeb sid new-vd.deb diff --git a/ROADMAP b/ROADMAP index 2f02febba..4c5e39d80 100644 --- a/ROADMAP +++ b/ROADMAP @@ -1,42 +1,67 @@ -# Feature Roadmap - -## Post 1.0 -- Python expressions should be usable like regex for search, select, etc -- delete regex match in cells -- maintain original ordering with implicit/hidden line# column -- popup windows for enum selection -- save as .json -- simple file management in VSheetDir: - - rename with 'e'dit - - mass rename with transform - - ^Delete - - only apply changes with ^S -- checkpoint data for replay/undo -- connect to larger datasets - - load larger than memory .csv/tsv files -- load new .py plugins for inputs/outputs/sheets - - git app that can streamline standard workflows - - checkout branch or revision with TUI interface - - browse through history -- '.' repeat edit? -- save all sheets to .h5 file -- reload derived sheets -- html table parsing into columns -- parse .xml -- how to handle timezone for datetime +# Roadmap + +I. Interface +- statusline menu for chooseOne +- context-sensitive hierarchical menu system to discover commands +- clickable column headers to set type/name, click-drag to move columns/rows +- keycols still pinned on display left (or maybe a disp_ option, which possibly affects col save order), but are not moved within columns list + +II. Internals +- break VisiData object into several smaller pseudo-classes, which can be plug-n-play. + - maybe provide a null functional implementation (raw interface). + - Progress/async can be completely standalone +- Sheet gets split into [Base]Sheet and Table + - Sheet is interface with draw, reload, exec_command + - Table inherits from Sheet, has rows/columns + - Plotter inherits from Sheet + - SourceSheet inherits from Table + - Enter as an alias for 'e' on SourceSheet + - newRow() (for 'a') on SourceSheet only +- all commands implemented on Sheet or children + - global key bindings like 'h' to 'move-left', and then each Sheet defines the semantics of its move-left +- allow Table.rows to be an offline datastore + +II. Loaders +- pandas dataframe +- .xml +- .sas +- try fixed width first for unknown types, fallback to TextSheet if no columns detected + +III. Savers +- '^S' saves as any input format depending on file extension +- 'g^S' saves all sheets (or selected on SheetsSheets) to hdf5, xlsx, and other multisheet formats +- 'z^S' saves just this cell +- sqlite: allow in-memory edits, ^S to commit +- upload to google sheets +- save to .py: emit code to get current cell/row/sheet from source data +- save canvas to .svg + +IV. Individual sheets + A. DirSheet + - add cached `file` type column + - rename file with any 'set' (allow mass rename with transform) + - 'd'elete file + - but only apply changes with ^S + - '^O' to open current file in external program (probably $EDITOR) + +## new commands and features + + +- numeric binning for freq column and pivot table +- Python expressions usable like regex for search, select, etc +- add row number in hidden column on data sheets so that original (load) order can be restored +- handle timezone for datetime # Applications -- vdsh ('x' executes a shell command and creates a sheet from the output) - - vdtop (includes kill/killall/ps) - - vdls (rm/mv/cp/mount/find/grep/mkdir/rmdir/ln/touch) - - vdnet (netstat) - - vdproc (explore /proc filesystem with some added goodies) -- vdgit -- vdb (interface to gdb) -- vdchat -- vdmail -- vdsearch +- [vgit](http://github.com/saulpw/vgit) (currently v0.2) +- [vsh](http://github.com/saulpw/vsh) (unreleased) + - vtop (includes kill/killall/ps) + - vls (rm/mv/cp/mount/find/grep/mkdir/rmdir/ln/touch) + - vnet (netstat) + - vping + - vproc (explore /proc filesystem with some added goodies) +- vpy (python explorer/debugger) +- vchat - [vdgalcon](http://github.com/saulpw/vdgalcon) -- vdscrape -- vdcalc +- vscrape diff --git a/bin/vd b/bin/vd index 8ca97b379..753dee359 100755 --- a/bin/vd +++ b/bin/vd @@ -3,7 +3,7 @@ # Usage: $0 [] [ ...] # $0 [] --play [--batch] [-w ] [-o ] [field=value ...] -__version__ = 'saul.pw/VisiData v0.98.1' +__version__ = 'saul.pw/VisiData v0.99' import os from visidata import * @@ -18,7 +18,7 @@ def eval_vd(logpath, *args, **kwargs): 'Instantiate logpath with args/kwargs replaced and replay all commands.' log = logpath.read_text().format(*args, **kwargs) src = PathFd(logpath.fqpn, io.StringIO(log), filesize=len(log)) - vs = openSource(src) + vs = openSource(src, filetype='vd') vd().push(vs) vs.vd = vd() return vs @@ -51,6 +51,7 @@ def main(): parser.add_argument('-w', dest='replay_wait', default=0, help='time to wait between replayed commands, in seconds') parser.add_argument('-d', dest='delimiter', help='delimiter to use for tsv filetype') parser.add_argument('--diff', dest='diff', default=None, help='show diffs from all sheets against this source') + parser.add_argument('-v', '--version', action='version', version=__version__) for optname, v in vdtui.baseOptions.items(): name, optval, defaultval, helpstr = v diff --git a/debian/README.Debian b/debian/README.Debian new file mode 100644 index 000000000..627c510c9 --- /dev/null +++ b/debian/README.Debian @@ -0,0 +1,5 @@ +VisiData for Debian + +This is the debian package for VisiData v0.99. It contains support for tsv, csv, fixed width, sqlite, json, xls, xlsx, http, html and graphs. + + -- Anja Boskovic Sat, 16 Dec 2017 15:41:18 -0500 diff --git a/debian/changelog b/debian/changelog new file mode 100644 index 000000000..726f42518 --- /dev/null +++ b/debian/changelog @@ -0,0 +1,14 @@ +visidata (0.99-1) unstable; urgency=low + + [Anja Boskovic] + * Bump to + [v0.99](https://github.com/saulpw/visidata/blob/stable/CHANGELOG.md#v099-2017-12-22) + + -- Anja Boskovic Wed, 20 Dec 2017 23:26:05 -0500 + +visidata (0.98.1-1) unstable; urgency=low + + [ Anja Boskovic ] + * Initial release. Closes: #884565 + + -- Anja Boskovic Sat, 16 Dec 2017 17:48:05 -0500 diff --git a/debian/compat b/debian/compat new file mode 100644 index 000000000..ec635144f --- /dev/null +++ b/debian/compat @@ -0,0 +1 @@ +9 diff --git a/debian/control b/debian/control new file mode 100644 index 000000000..78599f29f --- /dev/null +++ b/debian/control @@ -0,0 +1,16 @@ +Source: visidata +Section: devel +Priority: extra +Maintainer: Anja Boskovic +Build-Depends: debhelper (>=9), dh-python, python3-all, python3-setuptools, python3-xlrd, python3-openpyxl +Standards-Version: 3.9.7 +Homepage: https://pypi.python.org/pypi/visidata +X-Python3-Version: >= 3.4 + +Package: visidata +Architecture: all +Multi-Arch: foreign +Depends: ${misc:Depends}, ${python3:Depends} +Description: rapidly explore columnar data in the terminal + VisiData is a terminal utility for exploring, arranging + and analysing tabular data. diff --git a/debian/copyright b/debian/copyright new file mode 100644 index 000000000..073a9de81 --- /dev/null +++ b/debian/copyright @@ -0,0 +1,11 @@ +Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ +Upstream-Name: VisiData +Upstream-Contact: Saul Pwanson +Source: https://github.com/saulpw/visidata + +Files: * +Copyright: 2017 Saul Pwanson +License: GPL-3+ + On Debian systems, the full text of the GNU General Public + License version 3 can be found in the file + '/usr/share/common-licenses/GPL-3'. diff --git a/debian/manpages b/debian/manpages new file mode 100644 index 000000000..bc009c06f --- /dev/null +++ b/debian/manpages @@ -0,0 +1 @@ +visidata/man/vd.1 diff --git a/debian/patches/series b/debian/patches/series new file mode 100644 index 000000000..e69de29bb diff --git a/debian/rules b/debian/rules new file mode 100755 index 000000000..2102f7621 --- /dev/null +++ b/debian/rules @@ -0,0 +1,7 @@ +#!/usr/bin/make -f + +export DH_VERBOSE=1 +export PYBUILD_NAME=visidata + +%: + dh $@ --with python3 --buildsystem=pybuild diff --git a/debian/source.lintian-overrides b/debian/source.lintian-overrides new file mode 100644 index 000000000..f5ef25065 --- /dev/null +++ b/debian/source.lintian-overrides @@ -0,0 +1 @@ +visidata source: source-is-missing diff --git a/debian/source/format b/debian/source/format new file mode 100644 index 000000000..163aaf8d8 --- /dev/null +++ b/debian/source/format @@ -0,0 +1 @@ +3.0 (quilt) diff --git a/debian/source/local-options b/debian/source/local-options new file mode 100644 index 000000000..e69de29bb diff --git a/debian/watch b/debian/watch new file mode 100644 index 000000000..6c9716cf4 --- /dev/null +++ b/debian/watch @@ -0,0 +1,3 @@ +version=3 +opts=uversionmangle=s/(rc|a|b|c)/~$1/ \ +https://pypi.debian.net/visidata/visidata-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz))) diff --git a/docs/RELEASE.md b/docs/RELEASE.md deleted file mode 100644 index 3c02769e6..000000000 --- a/docs/RELEASE.md +++ /dev/null @@ -1,93 +0,0 @@ -# Release process for the next `stable` version - -1. Merge `stable` to `develop` (if necessary) - -2. Ensure `develop` automated tests run correctly with ./test.sh - -3. Verify that documentation/docstrings are up-to-date on features and functionality - -4. Set version number to next most reasonable number (v#.#.#) - - a. add to front of CHANGELOG, along with the release date and bullet points of major changes - - b. update version number on README and front page of website - - c. bump version in `__version__` in source code (bin/vd, visidata/vdtui.py) and setup.py - -5. Run ./mkwww.sh to build the manpage and updated website - -6. Push `develop` to testpypi - - a. set up a ~/.pypirc - - ``` - [distutils] - index-servers= - pypi - testpypi - [pypi] - repository:https://upload.pypi.org/legacy/ - username: - password: - - [testpypi] - repository: https://test.pypi.org/legacy - username: - password: - ``` - - b. push to testpypi - - ``` - python3 setup.py sdist - twine upload dist/* -r testpypi - ``` - -7. Test install from testpypi - - a. on virgin instance - - b. on mac and linux - - c. See if windows works - - d. from git clone - - ``` - pip3 install --extra-index-url https://test.pypi.org/project visidata - ``` - -10. Merge `develop` to stable - -11. Merge `stable` back into other branches - - a. if the branch works with minimal conflicts, keep the branch - - b. otherwise, clean out the branch - - -12. Push stable to pypi - -``` -twine upload dist/* -``` - -13. Test install/upgrade from pypi - - a. build and check readthedocs/stable - - b. Ask someone else to test install - -14. Create a tag `v#.#.#` for that commit -``` -git tag v#.#.# -git push --tags -``` - -15. Push code to stable - -16. Write up the release notes and post at visidata.org/release/#.# - -17. Comb through issues and close the ones that have been solved, referencing the version number - -18. Post release notes on r/visidata and tinyletter and have some ice cream diff --git a/docs/architecture.rst b/docs/architecture.rst deleted file mode 100644 index a3a945674..000000000 --- a/docs/architecture.rst +++ /dev/null @@ -1,590 +0,0 @@ -==================================== -VisiData Architecture for Developers -==================================== - -VisiData is like a powerful spreadsheet from an alternate textpunk reality, in -which data can be easily manipulated from the keyboard and terminal. Unlike a -spreadsheet, however, the data is well-structured, so that the data model is -closer to an RDBMS. - -* The main unit of functionality is the *sheet*. - -* Sheets have *rows* and *columns*. - -* Each sheet has a homogeneous list of rows, which can be any kind of Python - object. - -* Individual cells do not contain arbitrary values, but are extracted by the - column from the particular Python object for that row. - -Constraining the data to fit within this architecture simplifies the -implementation and allows for some radical optimizations to data workflow. - -The process of designing a sheet is: - -1. Instantiate the sheet from a toplevel command (or other sheet); -2. Collect the rows from the sources in reload(); -3. Enumerate the available columns; -4. Create commands to interact with the rows, columns, and cells. -5. Try the resulting workflow and iterate until it feels like magic. - -Columns -======= - -Note that each ``Column`` object is detached from any sheets in which it -appears. Think of it as a lens through which each individual *row* of a sheet -is viewed. Every ``Column`` must have at least a ``name`` and a ``getter`` method. - -``name`` and other properties ------------------------------ - -Columns have a few properties, all optional in the constructor except for ``name``: - -* **name**: should be a valid Python identifier and unique among - the column names on the sheet. Some features may not work if these conditions - are not met. - -* **type**: defaults to ``str``; other values are ``int``, ``float``, - ``date``, ``currency``. There is also a dummy ``anytype`` to produce a - stringified version for anything not in these categories. - -* **width**: specifies the default width for the column; ``0`` means - hidden. - -* **fmtstr**: format string for use with ``type`` when ``type`` is a date. - -* **aggregators**: a dictionary providing a few simple statistical - functions (``sum``, ``mean``, ``max``, etc.). - -* **expr**: Python expression that generates values if the column is a - "computed column". - - -Getter and setter ------------------ - -Each ``Column`` object has ``getter`` and ``setter`` methods; both are lambdas. -These lambdas are the "lenses" mentioned above — they are used on the fly to -display the cells of each row that (apparently) intersects with the column. - -Getter -~~~~~~ - -This lambda function is required. It takes a row as input and returns the value -for that column. This is the essential functionality of a ``Column``. - -A ``getter`` has wrapper methods ``getValue`` and ``getDisplayValue`` to -represent a value as its declared type or to format a value properly for -display. - -Setter -~~~~~~ - -The ``setter`` lambda function allows a row to be modified by the user using -the ``Sheet.editCell`` method. It takes a row object and new value, and sets -the value for that column. - -When a new ``Column`` object is initialized, ``setter`` defaults to ``None``, -making the column read-only. - -``Column.setValues`` is used to set the values for one or many rows programmatically. - -Built-in methods for column-creation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are several helper methods for constructing a ``Column`` object: - -* ``ColumnAttr(attrname, **kwargs)`` gets/sets an attribute from the row object using - ``getattr``/``setattr``. - This is useful when the rows are Python objects. - -* ``ColumnItem(colname, itemkey, **kwargs)`` uses ``__getitem__``/``setitem`` on the row. - This is useful when the rows are Python mappings, like dict. - -* ``SubrowColumn(origcol, subrowidx, **kwargs)`` uses ``origcol.getter(row[i])``. This is useful for rows which are a list of references to other rows, like with joined sheets. - ----- - -Commands -======== - -Keyboard commands are the primary interface for the user to control VisiData. -Add new commands using the global ``globalCommand()`` or ``Command()`` functions within a .py file. - -Syntax ------- - -``command()`` takes three arguments: - -* *command sequence*: the sequence of keys pressed to trigger the action. (Note - that if the control-key is involved, control is represented by ``^`` and the - following key must be upper-case. This is a stricture of Curses.) - -* *exec string*: a string containing valid python code that will be passed to - ``exec``. This string is limited to a single line of Python; longer code must - be placed in a separate "add-on" module (see `Extending VisiData`_). - -* *help string*: help text provided to users on the help sheet. - -Example -------- - -For example, VisiData has a builtin command ``Shift-P`` to take a random sample -of rows from the current sheet: - -:: - - command('P', - 'vd.push(sheet.copy("_sample")).rows= - random.sample(rows, int(input("random population size: ")))', - 'push duplicate sheet with a random sample of rows') - -Here the command sequence is regular ASCII ``P``, but it could include one or -more prefixes or consist of a Curses `key constant -`_ (e.g. -``KEY_HOME``). - -The ``exec`` string in this example illustrates the basic interface for -commands. Below we dissect various elements in the example. - -* The global ``VisiData`` singleton object is available as ``vd`` in the exec - string (and ``vd()`` in other contexts). - -* The ``VisiData.push`` method pushes a ``Sheet`` object onto the ``sheets`` - stack, making it the currently visible sheet. It returns that same sheet, so - that a member (in this case, ``rows``) may be conveniently set without using - a temporary variable. - -* The current sheet is available as ``sheet``. - -* The current sheet is also passed as the locals dict to ``exec``, so all Sheet - members and methods can be read and called without referencing ``sheet`` - explicitly. **Note**: due to the implementation of ``Sheet.exec_command``, - setting sheet members requires ``sheet`` to be passed explicitly. That is, - when a sheet member variable is on the LHS of an assignment, it must be - referred to as ``sheet.member`` or the assignment will not stick. - -* The ``Sheet.copy`` member function takes a string, which is appended to the - original sheet name to make the new sheet's name. - -* ``random.sample`` is a builtin Python function. The ``random`` package is - imported by VisiData (and thus available to all extensions automatically); - other packages may be imported at the toplevel of the .py extension. - -* ``input`` is a global function that displays a prompt and gets a string of - input from the user (on the bottom line). - -What can be done with commands ------------------------------- - -Anything is possible! However, the ``exec`` string limits functionality to -Python one-liners. More complicated commands require a custom sheet ("add-on") -to implement longer Python functions. - -There will eventually be a VisiData API reference. In the meantime, please see -the source code for examples of how to accomplish most tasks. - ----- - -Extending VisiData -================== - -Extend VisiData by defining custom sheets, in an "add-on". An add-on is a -non-core Python module, available to VisiData if placed in ``visidata/addons`` -and given a top-level key-binding that is available on all sheets. The add-on -returns specialized ``Sheet`` objects which are pushed onto the -``VisiData.sheets`` stack, initiated by a top-level command available on all -sheets. - -Outline of syntax ------------------ - -The skeleton of an add-on, apart from its actual functionality, is as follows: - -* Subclass ``Sheet``. In ``__init__``: - - * Add a command (using ``command()``) that instantiates the class and pushes - it onto a ``vd`` instance. You may also like to add options, using the - ``option`` command - - * Call ``super`` to define the name of the new sheet. - - * The constructor passes the name of the sheet and any source sheets - (available later as ``Sheet.source``). - - * Populate columns ``self.columns`` with a list of all possible columns. - Each entry should be a ``Column`` object (or subclass) and should have a - name. - - * Define any sheet-specific commands, using ``self.command()`` within the - constructor. The arguments are identical to those of the global - ``command()`` function (see `Commands`_). - -* Define ``reload`` to as to recompute the values of the rows. See - `reload()`_ below. - -* Consider whether the sheet may be so large or slow to recompute that you - don't want to user to be blocked waiting for reloading to finish. Some - sheets, such as the help sheet, cannot become that large and so there is - no need for asynchronous handling. But if it may become large, then: - - * Use ``Progress`` to display a progress bar showing the percentage of - rows recomputed. - - * Decorate ``reload`` with `@async`_. - -Example -~~~~~~~ - -Here is a simple sheet which makes a ``t`` command to "take" the current -cell from any sheet and append it to a predefined "journal" sheet. This -sheet can be viewed with ``Shift-T`` and then dumped to a ``.tsv`` file with -``Ctrl-w``. - -:: - - from visidata import * - - command('t', - 'vd.journal.rows.append([sheet, cursorCol, cursorRow])', - 'take this cell and append it to the journal') - command('T', 'vd.push(vd.journal)', 'push the journal') - - option('fn_journal', 'journal.tsv', 'default journal output file') - - class JournalSheet(Sheet): - def __init__(self): - super().__init__('journal') - - self.columns = [ - Column('sheet', getter=lambda r: r[0].name), - Column('column', getter=lambda r: r[1].name), - Column('value', getter=lambda r: r[1].getValue(r[2])), - ] - - self.command('^W', - 'appendToJournalFile(); sheet.rows = []', - 'append to existing journal and clear sheet') - - def appendToJournalFile(self): - p = Path(options.fn_journal) - writeHdr = not p.exists() - - with p.open_text('a') as fp: - if writeHdr: - fp.write('\t'.join('sheet', 'column', 'value')) - status('created journal at %s' % str(p)) - for r in self.rows: - fp.write('\t'.join(col.getDisplayValue(r) - for col in self.columns) + '\n') - status('saved %d rows' % len(self.rows)) - - vd().journal = JournalSheet() - -Note that the ``t`` command includes ``cursorRow`` in the list instead of -``cursorValue``, and when the journal is saved the value in the column of -the referenced row is retrieved using ``Column.getValue``. This is the -desired pattern for appending rows based on existing sheets, so that -changes to the source row are automatically reflected in the subsheets. - -Custom VisiData applications ----------------------------- - -Import the ``visidata`` package into a Python script to create a custom -VisiData application. - -For an example, see `vdgalcon `_: ``Ctrl-a`` for -start of line, ``Ctrl-e`` for end of line, -and so on. One innovation is ``Ctrl-r`` to reload the initial value of a cell. - -Module-level ``editText`` is wrapped by ``VisiData.editText`` and -``Sheet.editCell``. - -Regular expressions (RegEx) ---------------------------- - -Developers may enjoy using regular expressions (RegEx) to select rows. -``VisiData.searchRegex`` is available for that purpose. The flavor of RegEx is -that of `Python `_, similar to that -of Perl rather than that of ``vi``. - -Drawing -------- - -(Not yet documented. Topics include ``colLayout`` and ``visibleCols``.) - -Colorizing ----------- - -Control of the colors of foreground and background text is in need of work and -is not yet documented. - -Theme colors and characters ---------------------------- - -(Not yet documented.) - -Making VisiData apps --------------------- - -(Not yet documented. Topics include ``set_global`` and the helper sheets -``TextSheet`` and ``DirSheet``.) - -Making VisiData sources ------------------------ - -(Not yet documented. Topics include ``Path`` objects, ``openSource``, and -``open_*``.) - ----- - -Common variables -================ - -Following are some variable names used frequently in the codebase, together with their usual associations: - - * ``c``: column - - * ``expr``: Python expression - - * ``D``, ``d``: dict - - * ``f``: function - - * ``fn``: filename - - * ``i``: target variable of iterator or generator - - * ``idx``: index - - * ``L``: list - - * ``p``: path - - * ``pv``: present value - - * ``r``: row - - * ``ret``: return value - - * ``rng``: range - - * ``s``: string - - * ``scr``: "screen" object in Curses - - * ``v``: name of variable - - * ``vd``: ``visidata.Visidata``, normally constructed as a singleton (one-time-only instance) as ``VisiData()`` - - * ``vs``: sheet, constructed as ``visidata.Sheet(name, path)`` or returned from some function as ``openURL(path)``, ``open_tsv(path)``, ``DirSheet(name, path)``, etc. - - * ``w``: width - - * ``x``: horizontal position on the screen - - * ``y``: vertical position on the screen - ----- - - -Unresolved hacks -================ - -Your insight as to how to improve these is most welcome. - -``chooseOne`` -------------- - -``chooseOne`` should be a proper chooser. - -Adding properties to ``vd`` in extensions ------------------------------------------ - -Adding a property to the VisiData singleton in an extension is done as in - ``visidata/status_history.py``: - - .. code-block:: python - - vd().statusHistory = [] - - -Globals -------- - -Accessing all commands in an extension requires the use of globals. The extension requires a statement like this for all importers. - - .. code-block:: python - - addGlobals(globals()) - - -Deviations from PEP8 --------------------- - -* One-line docstrings are surrounded by a single quote (``'...'``). - -* Multi-line docstrings are surrounded by three single quotes (``'''...'''``). - -* Names of functions and variables are mostly in camel case, with some exceptions. - - - diff --git a/docs/ascii-commands.txt b/docs/ascii-commands.txt index b18354173..7eb51d27f 100644 --- a/docs/ascii-commands.txt +++ b/docs/ascii-commands.txt @@ -7,9 +7,9 @@ Ctrl- F Forward one page [vim] Ctrl- G file information [vim] Ctrl- H - Ctrl- I + Ctrl- I TAB (advance replay) Ctrl- J ENTER (dive into row/value) [sheet-specific] - Ctrl- K + Ctrl- K abort replay Ctrl- L redraw screen Ctrl- M (must be same as ENTER) Ctrl- N @@ -18,18 +18,18 @@ Ctrl- Q Quit Ctrl- R Reload Ctrl- S Save - Ctrl- T Tasks sheet + Ctrl- T Threads sheet Ctrl- U paUse/resUme Ctrl- V Version Ctrl- W Ctrl- X push sheet with eval of python e'X'pression Ctrl- Y push sheet for this row object - Ctrl- Z + Ctrl- Z send SIGSTOP Ctrl- [ ESC Ctrl- \ Ctrl- ] Ctrl- ^ swap sheets [vim] - Ctrl- _ + Ctrl- _ toggle profile of main interface thread SPACE advance commandlog replay to next step Shift- ! toggle key column Shift- " duplicate sheet (keeping only selected rows) @@ -44,7 +44,7 @@ Shift- + set column aggregator , select matching rows ['pick up', from nethack] - hide column - . + . plot / search forward 0 1 @@ -116,7 +116,7 @@ s select row t toggle row select u unselect row - v + v visibility toggle w x execute this row (meaning varies depending on sheet) y yank a row to clipbard [vim] diff --git a/docs/img/birdsdiet_bymass.gif b/docs/img/birdsdiet_bymass.gif deleted file mode 100644 index c4e682427..000000000 Binary files a/docs/img/birdsdiet_bymass.gif and /dev/null differ diff --git a/docs/img/screenshot.gif b/docs/img/screenshot.gif deleted file mode 100644 index 8d79bed96..000000000 Binary files a/docs/img/screenshot.gif and /dev/null differ diff --git a/docs/img/tour01/01.jpg b/docs/img/tour01/01.jpg deleted file mode 100644 index 8aace2e0a..000000000 Binary files a/docs/img/tour01/01.jpg and /dev/null differ diff --git a/docs/img/tour01/02.jpg b/docs/img/tour01/02.jpg deleted file mode 100644 index 17cf4bade..000000000 Binary files a/docs/img/tour01/02.jpg and /dev/null differ diff --git a/docs/img/tour01/03.jpg b/docs/img/tour01/03.jpg deleted file mode 100644 index c7363db29..000000000 Binary files a/docs/img/tour01/03.jpg and /dev/null differ diff --git a/docs/img/tour01/04.jpg b/docs/img/tour01/04.jpg deleted file mode 100644 index 658455eff..000000000 Binary files a/docs/img/tour01/04.jpg and /dev/null differ diff --git a/docs/img/tour01/05.jpg b/docs/img/tour01/05.jpg deleted file mode 100644 index 1011db6bf..000000000 Binary files a/docs/img/tour01/05.jpg and /dev/null differ diff --git a/docs/img/tour01/06.jpg b/docs/img/tour01/06.jpg deleted file mode 100644 index a102fa4a8..000000000 Binary files a/docs/img/tour01/06.jpg and /dev/null differ diff --git a/docs/img/tour01/07.jpg b/docs/img/tour01/07.jpg deleted file mode 100644 index 2bbc45c21..000000000 Binary files a/docs/img/tour01/07.jpg and /dev/null differ diff --git a/docs/img/tour01/08.jpg b/docs/img/tour01/08.jpg deleted file mode 100644 index 4436ee8b7..000000000 Binary files a/docs/img/tour01/08.jpg and /dev/null differ diff --git a/docs/img/tour01/09.jpg b/docs/img/tour01/09.jpg deleted file mode 100644 index 4471c02bf..000000000 Binary files a/docs/img/tour01/09.jpg and /dev/null differ diff --git a/docs/img/tour01/10.jpg b/docs/img/tour01/10.jpg deleted file mode 100644 index b95da2599..000000000 Binary files a/docs/img/tour01/10.jpg and /dev/null differ diff --git a/docs/img/tour01/11.jpg b/docs/img/tour01/11.jpg deleted file mode 100644 index 0bf0a4607..000000000 Binary files a/docs/img/tour01/11.jpg and /dev/null differ diff --git a/docs/img/tour01/12.jpg b/docs/img/tour01/12.jpg deleted file mode 100644 index 9750193d0..000000000 Binary files a/docs/img/tour01/12.jpg and /dev/null differ diff --git a/docs/img/tour01/13.jpg b/docs/img/tour01/13.jpg deleted file mode 100644 index 89d7b96a0..000000000 Binary files a/docs/img/tour01/13.jpg and /dev/null differ diff --git a/docs/img/tour01/14.jpg b/docs/img/tour01/14.jpg deleted file mode 100644 index be37aa4ea..000000000 Binary files a/docs/img/tour01/14.jpg and /dev/null differ diff --git a/docs/img/tour01/15.jpg b/docs/img/tour01/15.jpg deleted file mode 100644 index 365d3631c..000000000 Binary files a/docs/img/tour01/15.jpg and /dev/null differ diff --git a/docs/img/tour01/16.jpg b/docs/img/tour01/16.jpg deleted file mode 100644 index 6d5562dae..000000000 Binary files a/docs/img/tour01/16.jpg and /dev/null differ diff --git a/docs/img/tour01/17.jpg b/docs/img/tour01/17.jpg deleted file mode 100644 index 6d30e1bb2..000000000 Binary files a/docs/img/tour01/17.jpg and /dev/null differ diff --git a/docs/img/tour01/18.png b/docs/img/tour01/18.png deleted file mode 100644 index aeadf312b..000000000 Binary files a/docs/img/tour01/18.png and /dev/null differ diff --git a/docs/img/tour01/19.jpg b/docs/img/tour01/19.jpg deleted file mode 100644 index f38962581..000000000 Binary files a/docs/img/tour01/19.jpg and /dev/null differ diff --git a/docs/index.md b/docs/index.md deleted file mode 100644 index 7ecdf2004..000000000 --- a/docs/index.md +++ /dev/null @@ -1 +0,0 @@ -Documentation for VisiData is now hosted [here](http://visidata.org). diff --git a/docs/tours.rst b/docs/tours.rst deleted file mode 100644 index e2795cdff..000000000 --- a/docs/tours.rst +++ /dev/null @@ -1,74 +0,0 @@ -============= -VisiData Tour -============= - -Example query: using `birdsdiet.tsv `_, what is the relationship between the diet and average mass of passerine birds? - -.. image:: img/birdsdiet_bymass.gif - :alt: VisiData Tour - -+----------------------------------------------------------------+--------------------------------------------------------------------+ -|**1. Open the data at the terminal with** ``vd birdsdiet.tsv`` | .. image:: img/tour01/01.jpg | -+----------------------------------------------------------------+--------------------------------------------------------------------+ -+---------------------------------------------------------------------------+---------------------------------------------------------+ -|**2. Toggle the** ``Family`` **column to be a key column with** ``!`` | .. image:: img/tour01/02.jpg | -+---------------------------------------------------------------------------+---------------------------------------------------------+ -+--------------------------------------------------------------+----------------------------------------------------------------------+ -|**3. Hide undesired columns with** ``-`` | .. image:: img/tour01/03.jpg | -+--------------------------------------------------------------+----------------------------------------------------------------------+ -+------------------------------------------------------------------+------------------------------------------------------------------+ -|**4. Set the** ``Mass`` **column to the float type with** ``%`` | .. image:: img/tour01/04.jpg | -+------------------------------------------------------------------+------------------------------------------------------------------+ -+------------------------------------------------------------------+------------------------------------------------------------------+ -|**5. Go to the** ``Passerine`` **column with** ``l`` | .. image:: img/tour01/05.jpg | -+------------------------------------------------------------------+------------------------------------------------------------------+ -+---------------------------------------------------------------------------------------------------+---------------------------------+ -|**6. Select rows by regex where the** ``Passerine`` **column contains a** ``1`` **with** ``|1`` | .. image:: img/tour01/06.jpg | -| | | -|``|`` selects rows which match the given regex in the current column, to later filter rows in/out. | | -+---------------------------------------------------------------------------------------------------+---------------------------------+ -+------------------------------------------------------------------+------------------------------------------------------------------+ -|**7. Push a new sheet of only the selected rows with** ``"`` | .. image:: img/tour01/07.jpg | -+------------------------------------------------------------------+------------------------------------------------------------------+ -+----------------------------------------------------------------------+--------------------------------------------------------------+ -|**8. Go to the** ``Mass`` **column with** ``h`` | .. image:: img/tour01/08.jpg | -+----------------------------------------------------------------------+--------------------------------------------------------------+ -+---------------------------------------------------------------------------------------------+---------------------------------------+ -|**9. Choose to aggregate the average** ``Mass`` **with** ``+``, ``mean`` **and** ``ENTER`` | .. image:: img/tour01/09.jpg | -| | | -|``+`` sets the aggregation function for the current column, to be used by grouping actions. | | -+---------------------------------------------------------------------------------------------+---------------------------------------+ -+--------------------------------------------------------------------------+----------------------------------------------------------+ -|**10. Go to the** ``Diet`` **column (with** ``l`` **)** | .. image:: img/tour01/10.jpg | -+--------------------------------------------------------------------------+----------------------------------------------------------+ -+------------------------------------------------------------------------------------+------------------------------------------------+ -|**11. Push a frequency table sheet for the** ``Diet`` **column with** ``F`` | .. image:: img/tour01/11.jpg | -| | | -|This sheet adds an aggregated column for every column with an aggregation function. | | -+------------------------------------------------------------------------------------+------------------------------------------------+ -+--------------------------------------------------------------------------------+----------------------------------------------------+ -|**12. Minimize the** ``Diet`` **column to fit the longest value with** ``_`` | .. image:: img/tour01/12.jpg | -+--------------------------------------------------------------------------------+----------------------------------------------------+ -+------------------------------------------------------------------+------------------------------------------------------------------+ -|**13. Go to the rightmost column with** ``gl`` | .. image:: img/tour01/13.jpg | -+------------------------------------------------------------------+------------------------------------------------------------------+ -+----------------------------------------------------------------------+--------------------------------------------------------------+ -|**14. Sort the rows by descending** ``avg_Mass`` **with** ``]`` | .. image:: img/tour01/14.jpg | -+----------------------------------------------------------------------+--------------------------------------------------------------+ -+--------------------------------------------------------------------------+----------------------------------------------------------+ -|**15. Go to the** ``histogram`` **column (with** ``h`` **)** | .. image:: img/tour01/15.jpg | -+--------------------------------------------------------------------------+----------------------------------------------------------+ -+--------------------------------------------------------------------------+----------------------------------------------------------+ -|**16. Hide the** ``histogram`` **column (with** ``-`` **)** | .. image:: img/tour01/16.jpg | -+--------------------------------------------------------------------------+----------------------------------------------------------+ -+-----------------------------------------------------------------------------------------------+-------------------------------------+ -|**17. Save the current sheet to** ``.tsv`` **with** ``Ctrl-s`` **followed by** ``ENTER`` | .. image:: img/tour01/17.jpg | -+-----------------------------------------------------------------------------------------------+-------------------------------------+ -+------------------------------------------------------------------------------------------------+------------------------------------+ -|**18. View commandlog with** ``D`` | .. image:: img/tour01/18.png | -| | | -|The commandlog lists every action taken since the program started. | | -+------------------------------------------------------------------------------------------------+------------------------------------+ -+------------------------------------------------------------------+------------------------------------------------------------------+ -|**19. Exit VisiData with** ``gq`` | .. image:: img/tour01/19.jpg | -+------------------------------------------------------------------+------------------------------------------------------------------+ diff --git a/mkwww.sh b/mkwww.sh index e2c42b621..992ab068b 100755 --- a/mkwww.sh +++ b/mkwww.sh @@ -11,8 +11,11 @@ BUILDWWW=$BUILD/www MAN=$VD/visidata/man TEST=$WWW/test DESIGN=$WWW/design +HOWTODEV=$WWW/howto/dev NEWS=$WWW/news VIDEOS=$WWW/videos +HELP=$WWW/help +INSTALL=$WWW/install # Build directories mkdir -p $BUILD @@ -20,9 +23,12 @@ mkdir -p $BUILDWWW mkdir -p $BUILDWWW/man mkdir -p $BUILDWWW/test mkdir -p $BUILDWWW/docs -#mkdir -p $BUILDWWW/design +mkdir -p $BUILDWWW/design +mkdir -p $BUILDWWW/howto/dev mkdir -p $BUILDWWW/about mkdir -p $BUILDWWW/contributing +mkdir -p $BUILDWWW/help +mkdir -p $BUILDWWW/install mkdir -p $BUILDWWW/videos # Set up python and shell environment @@ -54,10 +60,28 @@ echo '' >> $BUILD/vd-man-inc.html # Properties of columns on the source sheet can be changed with standard editing commands (e $VD/strformat.py body=$BUILD/vd-man-inc.html title="VisiData Quick Reference" head="" < $WWW/template.html > $BUILDWWW/man/index.html +# Create http://visidata.org/man/#loaders +sed -i -e "s#SUPPORTED SOURCES#SUPPORTED SOURCES#g" $BUILDWWW/man/index.html + # Build /contributing pandoc -r markdown -w html -o $BUILDWWW/contributing/index.body $VD/CONTRIBUTING.md $VD/strformat.py body=$BUILDWWW/contributing/index.body title="Contributing to VisiData" head="" < $WWW/template.html > $BUILDWWW/contributing/index.html +# Build /help +pandoc -r markdown -w html -o $BUILDWWW/help/index.body $HELP/index.md +$VD/strformat.py body=$BUILDWWW/help/index.body title="Support" head="" < $WWW/template.html > $BUILDWWW/help/index.html + +# Build /install +pandoc -r markdown -w html -o $BUILDWWW/install/index.body $INSTALL/index.md +$VD/strformat.py body=$BUILDWWW/install/index.body title="Installation" head="" < $WWW/template.html > $BUILDWWW/install/index.html + +# Create http://visidata.org/install/#pip3 +sed -i -e "s#

Install via pip3

#

Install via pip3

#g" $BUILDWWW/install/index.html +# Create http://visidata.org/install/#brew +sed -i -e "s#

Install via brew

#

Install via brew

#g" $BUILDWWW/install/index.html +# Create http://visidata.org/install/#apt +sed -i -e "s#

Install via apt

#

Install via apt

#g" $BUILDWWW/install/index.html + # Build /videos $VD/strformat.py body=$VIDEOS/video-body.html title="VisiData Videos" head="" < $WWW/template.html > $BUILDWWW/videos/index.html @@ -80,18 +104,29 @@ pandoc -r markdown -w html -o $BUILDWWW/docs/index.body $WWW/docs.md $VD/strformat.py body=$BUILDWWW/docs/index.body title="VisiData documentation" head="" < $WWW/template.html > $BUILDWWW/docs/index.html # Build /design -#pandoc -r markdown -w html -o $BUILDWWW/design/index.body $WWW/design.md -#$VD/strformat.py body=$BUILDWWW/design/index.body title="VisiData Design and Internals" head="" < $WWW/template.html > $BUILDWWW/design/index.html -#rm -f $BUILDWWW/design/index.body -#for postpath in `find $DESIGN -name '*.md'`; do -# post=${postpath##$DESIGN/} -# postname=${post%.md} -# mkdir -p $BUILDWWW/design/$postname -# posthtml=$BUILDWWW/design/$postname/index -# pandoc -r markdown -w html -o $posthtml.body $postpath -# $VD/strformat.py body=$posthtml.body title=$postname head="" < $WWW/template.html > $posthtml.html -# rm -f $posthtml.body -#done +pandoc -r markdown -w html -o $BUILDWWW/design/index.body $WWW/design.md +$VD/strformat.py body=$BUILDWWW/design/index.body title="VisiData Design and Internals" head="" < $WWW/template.html > $BUILDWWW/design/index.html +rm -f $BUILDWWW/design/index.body +for postpath in `find $DESIGN -name '*.md'`; do + post=${postpath##$DESIGN/} + postname=${post%.md} + mkdir -p $BUILDWWW/design/$postname + posthtml=$BUILDWWW/design/$postname/index + pandoc -r markdown -w html -o $posthtml.body $postpath + $VD/strformat.py body=$posthtml.body title=$postname head="" < $WWW/template.html > $posthtml.html + rm -f $posthtml.body +done + +# Build /howto/dev +for postpath in `find $HOWTODEV -name '*.md'`; do + post=${postpath##$HOWTODEV/} + postname=${post%.md} + mkdir -p $BUILDWWW/howto/dev/$postname + posthtml=$BUILDWWW/howto/dev/$postname/index + pandoc -r markdown -w html -o $posthtml.body $postpath + $VD/strformat.py body=$posthtml.body title=$postname head="" < $WWW/template.html > $posthtml.html + rm -f $posthtml.body +done # Build /news mkdir -p $BUILDWWW/news @@ -108,10 +143,8 @@ for postpath in `find $NEWS -name '*.md'`; do rm -f $posthtml.body done -# Build /help -mkdir -p $BUILDWWW/help -pandoc -r markdown -w html -o $BUILDWWW/help/index.body $VD/CONTRIBUTING.md -$VD/strformat.py body=$BUILDWWW/help/index.body title="Contributing to VisiData as a user or developer" head="" < $WWW/template.html > $BUILDWWW/help/index.html +# Add the key +cp $WWW/devotees.gpg.key $BUILDWWW #### At the end # add analytics to .html files diff --git a/requirements.txt b/requirements.txt index 679bb8355..fe9c1e437 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,8 +1,12 @@ -python-dateutil +# included in visidata +python-dateutil # date type +requests # http +lxml # html +openpyxl # xlsx +xlrd # xls -openpyxl -xlrd -h5py -psycopg2 -pyshp -mapbox-vector-tile +# included in visidata[full] +h5py # hdf5 +psycopg2 # postgres +pyshp # shapefiles +mapbox-vector-tile # mbtiles diff --git a/setup.py b/setup.py index c31f98a8d..9843a40e9 100755 --- a/setup.py +++ b/setup.py @@ -3,17 +3,18 @@ from setuptools import setup # tox can't actually run python3 setup.py: https://github.com/tox-dev/tox/issues/96 #from visidata import __version__ -__version__ = '0.98.1' +__version__ = '0.99' setup(name='visidata', version=__version__, - install_requires=['python-dateutil'], + install_requires=['python-dateutil', 'openpyxl', 'xlrd', 'lxml', 'requests'], extras_require={ - 'full': 'openpyxl xlrd h5py psycopg2 pyshp mapbox-vector-tile'.split() + 'full': 'h5py psycopg2 pyshp mapbox-vector-tile'.split() }, description='curses interface for exploring and arranging tabular data', long_description=open('README.md').read(), author='Saul Pwanson', + python_requires='>=3.4', author_email='visidata@saul.pw', url='http://visidata.org', download_url='https://github.com/saulpw/visidata/tarball/' + __version__, @@ -22,7 +23,7 @@ py_modules = ['visidata'], packages=['visidata', 'visidata.loaders'], include_package_data=True, - data_files = [('man/man1', ['visidata/man/vd.1'])], + data_files = [('share/man/man1', ['visidata/man/vd.1'])], package_data={'': ['man/vd.1']}, license='GPLv3', classifiers=[ diff --git a/test.sh b/test.sh index 13a6afc91..3f8f34abd 100755 --- a/test.sh +++ b/test.sh @@ -1,5 +1,7 @@ #!/usr/bin/env bash +trap "echo aborted; exit;" SIGINT SIGTERM + # Usage $0 for i in tests/*.vd ; do echo "--- $i" diff --git a/testdiff.sh b/testdiff.sh index abebd9436..133b2e6f3 100755 --- a/testdiff.sh +++ b/testdiff.sh @@ -1,10 +1,8 @@ #!/bin/bash -for fn in tests/*.vd ; do - if [ "${fn%-notest.vd}-notest" != "${fn%.vd}" ] +for fn in `git diff --name-only -- *.tsv` ; do + if [ "${fn%-notest.tsv}-notest" != "${fn%.tsv}" ] then - fna=${fn##tests/} - tsvfn=tests/golden/${fna%.vd}.tsv - git show HEAD^:$tsvfn | bin/vd --diff $tsvfn + git show HEAD^:$fn | bin/vd --diff $fn fi done diff --git a/tests/display-null.vd b/tests/display-null.vd index 512d4268e..002ef30be 100644 --- a/tests/display-null.vd +++ b/tests/display-null.vd @@ -1,8 +1,8 @@ sheet col row keystrokes input comment o sample_data/surveys.csv open file surveys record_id 0 O open Options -options value 15 e Null edit option -options value 15 ^^ jump to previous sheet (swaps with current sheet) +options value 16 e Null edit option +options value 16 ^^ jump to previous sheet (swaps with current sheet) surveys record_id 0 zr 24 move to the given row number surveys sex 24 , select rows matching current cell in current column surveys sex 24 gzd set contents of cells in current column to None for selected rows diff --git a/visidata-brew.rb b/visidata-brew.rb new file mode 100644 index 000000000..0bf3de39a --- /dev/null +++ b/visidata-brew.rb @@ -0,0 +1,57 @@ +class Visidata < Formula + include Language::Python::Virtualenv + desc "Terminal utility for exploring and arranging tabular data" + homepage "http://visidata.org" + url "https://github.com/saulpw/visidata/archive/v0.98.1.tar.gz" + sha256 "853d19ee2a74f986feeb3fc08656a349a4b26ee090b595d41f69bce777dd3907" + head "https://github.com/saulpw/visidata.git" + + depends_on :python3 + + resource "six" do + url "https://files.pythonhosted.org/packages/16/d8/bc6316cf98419719bd59c91742194c111b6f2e85abac88e496adefaf7afe/six-1.11.0.tar.gz" + sha256 "70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9" + end + + resource "python-dateutil" do + url "https://files.pythonhosted.org/packages/54/bb/f1db86504f7a49e1d9b9301531181b00a1c7325dc85a29160ee3eaa73a54/python-dateutil-2.6.1.tar.gz" + sha256 "891c38b2a02f5bb1be3e4793866c8df49c7d19baabf9c1bad62547e0b4866aca" + end + + resource "et-xmlfile" do + url "https://files.pythonhosted.org/packages/22/28/a99c42aea746e18382ad9fb36f64c1c1f04216f41797f2f0fa567da11388/et_xmlfile-1.0.1.tar.gz" + sha256 "614d9722d572f6246302c4491846d2c393c199cfa4edc9af593437691683335b" + end + + resource "jdcal" do + url "https://files.pythonhosted.org/packages/9b/fa/40beb2aa43a13f740dd5be367a10a03270043787833409c61b79e69f1dfd/jdcal-1.3.tar.gz" + sha256 "b760160f8dc8cc51d17875c6b663fafe64be699e10ce34b6a95184b5aa0fdc9e" + end + + resource "openpyxl" do + url "https://files.pythonhosted.org/packages/8c/75/c4e557207c7ff3d217d002d4fee32b4e5dbfc5498e2a2c9ce6b5424c5e37/openpyxl-2.4.9.tar.gz" + sha256 "95e007f4d121f4fd73f39a6d74a883c75e9fa9d96de91d43c1641c103c3a9b18" + end + + resource "xlrd" do + url "https://files.pythonhosted.org/packages/86/cf/bb010f16cefa8f26ac9329ca033134bcabc7a27f5c3d8de961bacc0f80b3/xlrd-1.1.0.tar.gz" + sha256 "8a21885513e6d915fe33a8ee5fdfa675433b61405ba13e2a69e62ee36828d7e2" + end + + def install + venv = virtualenv_create(libexec, "python3") + venv.pip_install resources + venv.pip_install_and_link buildpath + man1.install "visidata/man/vd.1" + Language::Python.each_python(build) do |_, version| + bundle_path = libexec/"lib/python#{version}/site-packages" + bundle_path.mkpath + ENV.prepend_path "PYTHONPATH", bundle_path + (lib/"python#{version}/site-packages/homebrew-visidata-bundle.pth").write "#{bundle_path}\n" + end + end + + test do + system "#{libexec}/bin/python3", "-c", "import visidata" + end +end diff --git a/visidata/Path.py b/visidata/Path.py index c860d88e8..a0aed53a4 100644 --- a/visidata/Path.py +++ b/visidata/Path.py @@ -76,12 +76,12 @@ def filesize(self): def __str__(self): return self.fqpn -class UrlPath: +class UrlPath(Path): def __init__(self, url): from urllib.parse import urlparse self.url = url self.obj = urlparse(url) - self.name = self.obj.netloc + super().__init__(self.obj.path) def __str__(self): return self.url @@ -121,6 +121,17 @@ def __enter__(self): def __exit__(self, a,b,c): pass + def read(self, n=None): + r = '' + while len(r) < n: + try: + s = next(self.iter) + r += s + '\n' + n += len(r) + except StopIteration: + break # end of file + return r + def seek(self, n): assert n == 0, 'RepeatFile can only seek to beginning' self.iter = RepeatFileIter(self) diff --git a/visidata/__init__.py b/visidata/__init__.py index 2ebc80fac..49735f33d 100644 --- a/visidata/__init__.py +++ b/visidata/__init__.py @@ -23,6 +23,7 @@ from .graph import * from .loaders.csv import * +from .loaders.json import * from .loaders.zip import * from .loaders.xlsx import * from .loaders.hdf5 import * @@ -31,5 +32,7 @@ from .loaders.postgres import * from .loaders.shp import * from .loaders.mbtiles import * +from .loaders.http import * +from .loaders.html import * addGlobals(globals()) diff --git a/visidata/aggregators.py b/visidata/aggregators.py index 7a1642125..f058acbe1 100644 --- a/visidata/aggregators.py +++ b/visidata/aggregators.py @@ -1,6 +1,6 @@ import collections from visidata import * -globalCommand('+', 'addAggregator([cursorCol], chooseOne(aggregators))', 'add aggregator to the current column') +globalCommand('+', 'addAggregator([cursorCol], chooseOne(aggregators))', 'add aggregator to current column') globalCommand('z+', 'status(chooseOne(aggregators)(cursorCol, selectedRows or rows))', 'display result of aggregator over values in selected rows for current column') aggregators = collections.OrderedDict() @@ -24,20 +24,21 @@ def mean(vals): if vals: return float(sum(vals))/len(vals) +def median(values): + L = sorted(values) + return L[len(L)//2] aggregator('min', min) aggregator('max', max) aggregator('avg', mean, float) aggregator('mean', mean, float) +aggregator('median', median) aggregator('sum', sum) aggregator('distinct', lambda values: len(set(values)), int) aggregator('count', lambda values: sum(1 for v in values), int) -def rowkeys(sheet, row): - return ' '.join(c.getDisplayValue(row) for c in sheet.keyCols) - # returns keys of the row with the max value -fullAggregator('keymax', anytype, lambda col, rows: rowkeys(col.sheet, max(col.getValueRows(rows))[1])) +fullAggregator('keymax', anytype, lambda col, rows: col.sheet.rowkey(max(col.getValueRows(rows))[1])) ColumnsSheet.commands += [ Command('g+', 'addAggregator(selectedRows or source.nonKeyVisibleCols, chooseOne(aggregators))', 'add aggregator to selected source columns'), diff --git a/visidata/async.py b/visidata/async.py index 9a9ac6fb4..b3cd874a5 100644 --- a/visidata/async.py +++ b/visidata/async.py @@ -5,14 +5,14 @@ from .vdtui import * -min_task_time_s = 0.10 # only keep tasks that take longer than this number of seconds +min_thread_time_s = 0.10 # only keep threads that take longer than this number of seconds -option('profile_tasks', True, 'profile async tasks') +option('profile_threads', True, 'profile async threads') option('min_memory_mb', 0, 'minimum memory to continue loading and async processing') -globalCommand('^C', 'cancelThread(*sheet.currentThreads or error("no active threads on this sheet"))', 'abort all tasks on current sheet') -globalCommand('g^C', 'cancelThread(*vd.threads or error("no threads"))', 'abort all secondary tasks') -globalCommand('^T', 'vd.push(vd.tasksSheet)', 'open Tasks Sheet') +globalCommand('^C', 'cancelThread(*sheet.currentThreads or error("no active threads on this sheet"))', 'abort all threads on current sheet') +globalCommand('g^C', 'cancelThread(*vd.threads or error("no threads"))', 'abort all secondary threads') +globalCommand('^T', 'vd.push(vd.threadsSheet)', 'open Threads Sheet') globalCommand('^_', 'toggleProfiling(threading.current_thread())', 'turn profiling on for main process') class ProfileSheet(TextSheet): @@ -27,10 +27,10 @@ def toggleProfiling(t): if not t.profile: t.profile = cProfile.Profile() t.profile.enable() - status('profiling of main task enabled') + status('profiling of main thread enabled') else: t.profile.disable() - status('profiling of main task disabled') + status('profiling of main thread disabled') # define @async for potentially long-running functions @@ -40,7 +40,7 @@ def toggleProfiling(t): class ThreadProfiler: def __init__(self, thread): self.thread = thread - if options.profile_tasks: + if options.profile_threads: self.thread.profile = cProfile.Profile() else: self.thread.profile = None @@ -55,13 +55,13 @@ def __exit__(self, exc_type, exc_val, tb): self.thread.profile.disable() # remove very-short-lived async actions - if elapsed_s(self.thread) < min_task_time_s: + if elapsed_s(self.thread) < min_thread_time_s: vd().threads.remove(self.thread) @functools.wraps(vd().toplevelTryFunc) def threadProfileCode(vdself, func, *args, **kwargs): - 'Profile @async tasks if `options.profile_tasks` is set.' + 'Profile @async threads if `options.profile_threads` is set.' with ThreadProfiler(threading.current_thread()) as prof: try: prof.thread.status = threadProfileCode.__wrapped__(vdself, func, *args, **kwargs) @@ -91,10 +91,10 @@ def cancelThread(*threads, exception=EscapeException): ] # each row is an augmented threading.Thread object -class TasksSheet(Sheet): +class ThreadsSheet(Sheet): rowtype = 'threads' commands = [ - Command('d', 'cancelThread(cursorRow)', 'abort task at current row'), + Command('d', 'cancelThread(cursorRow)', 'abort thread at current row'), Command('^C', 'd'), Command(ENTER, 'vd.push(ProfileSheet(cursorRow.name+"_profile", cursorRow.profile))', 'push profile sheet for this action'), ] @@ -124,7 +124,7 @@ def checkMemoryUsage(vs): attr = 'green' return ret, attr -vd().tasksSheet = TasksSheet('task_history') +vd().threadsSheet = ThreadsSheet('thread_history') vd().toplevelTryFunc = threadProfileCode vd().addHook('rstatus', checkMemoryUsage) diff --git a/visidata/canvas.py b/visidata/canvas.py index 4de8c506f..71b8dc639 100644 --- a/visidata/canvas.py +++ b/visidata/canvas.py @@ -2,164 +2,218 @@ from collections import defaultdict, Counter from visidata import * -# The Canvas covers the entire terminal (minus the status line). -# The Grid is arbitrarily large (a virtual cartesian coordinate system). -# The visibleGrid is what is actively drawn onto the terminal. -# The gridCanvas is the area of the canvas that contains that visibleGrid. -# The gridAxes are drawn outside the gridCanvas, on the left and bottom. - -# Canvas and gridCanvas coordinates are pixels at the same scale (gridCanvas are offset to leave room for axes). -# Grid and visibleGrid coordinates are app-specific units. - -# plotpixel()/plotline()/plotlabel() take Canvas pixel coordinates -# point()/line()/label() take Grid coordinates +# see www/design/graphics.md option('show_graph_labels', True, 'show axes and legend on graph') option('plot_colors', 'green red yellow cyan magenta white 38 136 168', 'list of distinct colors to use for plotting distinct objects') option('disp_pixel_random', False, 'randomly choose attr from set of pixels instead of most common') option('zoom_incr', 2.0, 'amount to multiply current zoomlevel by when zooming') +option('color_graph_hidden', '238 blue', 'color of legend for hidden attribute') + +class Point: + def __init__(self, x, y): + self.x = x + self.y = y + + def __str__(self): + if isinstance(self.x, int): + return '(%d,%d)' % (self.x, self.y) + else: + return '(%.02f,%.02f)' % (self.x, self.y) + + @property + def xy(self): + return (self.x, self.y) + +class Box: + def __init__(self, x, y, w=0, h=0): + self.xmin = x + self.ymin = y + self.w = w + self.h = h + + @property + def xymin(self): + return Point(self.xmin, self.ymin) + + @property + def xmax(self): + return self.xmin + self.w + + @property + def ymax(self): + return self.ymin + self.h + + @property + def center(self): + return Point(self.xcenter, self.ycenter) + + @property + def xcenter(self): + return self.xmin + self.w/2 + + @property + def ycenter(self): + return self.ymin + self.h/2 + + def contains(self, x, y): + return x >= self.xmin and \ + x < self.xmax and \ + y >= self.ymin and \ + y < self.ymax + +def BoundingBox(x1, y1, x2, y2): + return Box(min(x1, x2), min(y1, y2), abs(x2-x1), abs(y2-y1)) + + +def clipline(x1, y1, x2, y2, xmin, ymin, xmax, ymax): + 'Liang-Barsky algorithm, returns [xn1,yn1,xn2,yn2] of clipped line within given area, or None' + dx = x2-x1 + dy = y2-y1 + pq = [ + (-dx, x1-xmin), # left + ( dx, xmax-x1), # right + (-dy, y1-ymin), # bottom + ( dy, ymax-y1), # top + ] + + u1, u2 = 0, 1 + for p, q in pq: + if p < 0: # from outside to inside + u1 = max(u1, q/p) + elif p > 0: # from inside to outside + u2 = min(u2, q/p) + else: # p == 0: # parallel to bbox + if q < 0: # completely outside bbox + return None + + if u1 > u2: # completely outside bbox + return None + + xn1 = x1 + dx*u1 + yn1 = y1 + dy*u1 + + xn2 = x1 + dx*u2 + yn2 = y1 + dy*u2 + + return xn1, yn1, xn2, yn2 + +def iterline(x1, y1, x2, y2): + 'Yields (x, y) coords of line from (x1, y1) to (x2, y2)' + xdiff = abs(x2-x1) + ydiff = abs(y2-y1) + xdir = 1 if x1 <= x2 else -1 + ydir = 1 if y1 <= y2 else -1 + + r = max(xdiff, ydiff) + if r == 0: # point, not line + yield x1, y1 + else: + x, y = x1, y1 + i = 0 + while i <= r: + x += xdir * xdiff / r + y += ydir * ydiff / r + + yield x, y + i += 1 + + +def anySelected(vs, rows): + for r in rows: + if vs.isSelected(r): + return True -# pixels covering whole actual terminal # - width/height are exactly equal to the number of pixels displayable, and can change at any time. # - needs to refresh from source on resize -# - all x/y/w/h in PixelCanvas are pixel coordinates -# - override cursorPixelBounds to specify a cursor -class PixelCanvas(Sheet): +class Plotter(Sheet): + 'pixel-addressable display of entire terminal with (x,y) integer pixel coordinates' columns=[Column('')] # to eliminate errors outside of draw() commands=[ Command('^L', 'refresh()', 'redraw all pixels on canvas'), - Command('w', 'options.show_graph_labels = not options.show_graph_labels', 'toggle show_graph_labels'), + Command('v', 'options.show_graph_labels = not options.show_graph_labels', 'toggle show_graph_labels'), Command('KEY_RESIZE', 'refresh()', ''), ] def __init__(self, name, **kwargs): super().__init__(name, **kwargs) self.labels = [] # (x, y, text, attr) - self.disabledAttrs = set() + self.hiddenAttrs = set() self.needsRefresh = False self.resetCanvasDimensions() def resetCanvasDimensions(self): 'sets total available canvas dimensions' - self.canvasMinY = 0 - self.canvasMinX = 0 - self.canvasWidth = vd().windowWidth*2 - self.canvasHeight = (vd().windowHeight-1)*4 # exclude status line + self.plotwidth = vd().windowWidth*2 + self.plotheight = (vd().windowHeight-1)*4 # exclude status line # pixels[y][x] = { attr: list(rows), ... } - self.pixels = [[defaultdict(list) for x in range(self.canvasWidth)] for y in range(self.canvasHeight)] + self.pixels = [[defaultdict(list) for x in range(self.plotwidth)] for y in range(self.plotheight)] def plotpixel(self, x, y, attr, row=None): - self.pixels[round(y)][round(x)][attr].append(row) - - @staticmethod - def clipline(x1, y1, x2, y2, xmin, ymin, xmax, ymax): - 'Liang-Barsky algorithm' - dx = x2-x1 - dy = y2-y1 - pq = [ - (-dx, x1-xmin), # left - ( dx, xmax-x1), # right - (-dy, y1-ymin), # bottom - ( dy, ymax-y1), # top - ] - - u1, u2 = 0, 1 - for p, q in pq: - if p < 0: # from outside to inside - u1 = max(u1, q/p) - elif p > 0: # from inside to outside - u2 = min(u2, q/p) - else: # p == 0: # parallel to bbox - if q < 0: # completely outside bbox - return None - - if u1 > u2: # completely outside bbox - return None - - xn1 = x1 + dx*u1 - yn1 = y1 + dy*u1 - - xn2 = x1 + dx*u2 - yn2 = y1 + dy*u2 - - return xn1, yn1, xn2, yn2 + self.pixels[y][x][attr].append(row) def plotline(self, x1, y1, x2, y2, attr, row=None): - for x, y in self.iterline(x1, y1, x2, y2): - self.plotpixel(x, y, attr, row) - - @staticmethod - def iterline(x1, y1, x2, y2): - 'Draws onscreen segment of line from (x1, y1) to (x2, y2)' - xdiff = abs(x2-x1) - ydiff = abs(y2-y1) - xdir = 1 if x1 <= x2 else -1 - ydir = 1 if y1 <= y2 else -1 - - r = round(max(xdiff, ydiff)) - - if r == 0: # point, not line - yield x1, y1 - else: - x, y = x1, y1 - for i in range(r+1): - x += xdir * xdiff / r - y += ydir * ydiff / r - - yield x, y + for x, y in iterline(x1, y1, x2, y2): + self.plotpixel(round(x), round(y), attr, row) def plotlabel(self, x, y, text, attr): self.labels.append((x, y, text, attr)) def plotlegend(self, i, txt, attr): - self.plotlabel(self.canvasWidth-30, i*4, txt, attr) + self.plotlabel(self.plotwidth-30, i*4, txt, attr) @property - def cursorPixelBounds(self): - 'Returns pixel bounds of cursor as [ left, top, right, bottom ]' - return [ 0, 0, 0, 0 ] + def plotterCursorBox(self): + 'Returns pixel bounds of cursor as a Box. Override to provide a cursor.' + return Box(0,0,0,0) - def withinBounds(self, x, y, bbox): - left, top, right, bottom = bbox - return x >= left and \ - x < right and \ - y >= top and \ - y < bottom + @property + def plotterMouse(self): + return Point(self.mouseX*2, self.mouseY*4) def getPixelAttrRandom(self, x, y): 'weighted-random choice of attr at this pixel.' c = list(attr for attr, rows in self.pixels[y][x].items() - for r in rows if attr not in self.disabledAttrs) + for r in rows if attr not in self.hiddenAttrs) return random.choice(c) if c else 0 def getPixelAttrMost(self, x, y): 'most common attr at this pixel.' r = self.pixels[y][x] - c = sorted((len(rows), attr) for attr, rows in r.items() if attr not in self.disabledAttrs) - return c[-1][1] if c else 0 + c = sorted((len(rows), attr, rows) for attr, rows in r.items() if attr not in self.hiddenAttrs) + if not c: + return 0 + _, attr, rows = c[-1] + if anySelected(self.source, rows): + attr, _ = colors.update(attr, 8, 'bold', 10) + return attr - def togglePixelAttrs(self, attr): - if attr in self.disabledAttrs: - self.disabledAttrs.remove(attr) + def hideAttr(self, attr, hide=True): + if hide: + self.hiddenAttrs.add(attr) else: - self.disabledAttrs.add(attr) + self.hiddenAttrs.remove(attr) self.plotlegends() - def getRowsInside(self, x1, y1, x2, y2): - for y in range(y1, y2): - for x in range(x1, x2): + def rowsWithin(self, bbox): + 'return list of deduped rows within bbox' + ret = {} + for y in range(bbox.ymin, bbox.ymax+1): + for x in range(bbox.xmin, bbox.xmax+1): for attr, rows in self.pixels[y][x].items(): - for r in rows: - yield r + if attr not in self.hiddenAttrs: + for r in rows: + ret[id(r)] = r + return list(ret.values()) def draw(self, scr): if self.needsRefresh: - self.plotAll() + self.render() scr.erase() if self.pixels: - cursorBBox = self.cursorPixelBounds + cursorBBox = self.plotterCursorBox getPixelAttr = self.getPixelAttrRandom if options.disp_pixel_random else self.getPixelAttrMost for char_y in range(0, vd().windowHeight-1): # save one line for status for char_x in range(0, vd().windowWidth): @@ -186,8 +240,8 @@ def draw(self, scr): else: attr = 0 - if self.withinBounds(char_x*2, char_y*4, cursorBBox) or \ - self.withinBounds(char_x*2+1, char_y*4+3, cursorBBox): + if cursorBBox.contains(char_x*2, char_y*4) or \ + cursorBBox.contains(char_x*2+1, char_y*4+3): attr, _ = colors.update(attr, 0, options.color_current_row, 10) if attr: @@ -198,90 +252,77 @@ def draw(self, scr): clipdraw(scr, int(pix_y/4), int(pix_x/2), txt, attr, len(txt)) -# virtual display of arbitrary dimensions -# - x/y/w/h are always in grid units (units convenient to the app) -# - allows zooming in/out # - has a cursor, of arbitrary position and width/height (not restricted to current zoom) -class GridCanvas(PixelCanvas): +class Canvas(Plotter): + 'zoomable/scrollable virtual canvas with (x,y) coordinates in arbitrary units' rowtype = 'plots' - aspectRatio = None + aspectRatio = 0.0 leftMarginPixels = 10*2 rightMarginPixels = 6*2 topMarginPixels = 0 bottomMarginPixels = 2*4 # reserve bottom line for x axis - commands = PixelCanvas.commands + [ - Command('move-left', 'sheet.cursorGridMinX -= cursorGridWidth', ''), - Command('move-right', 'sheet.cursorGridMinX += cursorGridWidth', ''), - Command('move-down', 'sheet.cursorGridMinY += cursorGridHeight', ''), - Command('move-up', 'sheet.cursorGridMinY -= cursorGridHeight', ''), + commands = Plotter.commands + [ + Command('move-left', 'sheet.cursorBox.xmin -= cursorBox.w', ''), + Command('move-right', 'sheet.cursorBox.xmin += cursorBox.w', ''), + Command('move-down', 'sheet.cursorBox.ymin += cursorBox.h', ''), + Command('move-up', 'sheet.cursorBox.ymin -= cursorBox.h', ''), - Command('zh', 'sheet.cursorGridMinX -= charGridWidth', ''), - Command('zl', 'sheet.cursorGridMinX += charGridWidth', ''), - Command('zj', 'sheet.cursorGridMinY += charGridHeight', ''), - Command('zk', 'sheet.cursorGridMinY -= charGridHeight', ''), + Command('zh', 'sheet.cursorBox.xmin -= canvasCharWidth', ''), + Command('zl', 'sheet.cursorBox.xmin += canvasCharWidth', ''), + Command('zj', 'sheet.cursorBox.ymin += canvasCharHeight', ''), + Command('zk', 'sheet.cursorBox.ymin -= canvasCharHeight', ''), - Command('gH', 'sheet.cursorGridWidth /= 2', ''), - Command('gL', 'sheet.cursorGridWidth *= 2', ''), - Command('gJ', 'sheet.cursorGridHeight /= 2', ''), - Command('gK', 'sheet.cursorGridHeight *= 2', ''), + Command('gH', 'sheet.cursorBox.w /= 2', ''), + Command('gL', 'sheet.cursorBox.w *= 2', ''), + Command('gJ', 'sheet.cursorBox.h /= 2', ''), + Command('gK', 'sheet.cursorBox.h *= 2', ''), - Command('H', 'sheet.cursorGridWidth -= charGridWidth', ''), - Command('L', 'sheet.cursorGridWidth += charGridWidth', ''), - Command('J', 'sheet.cursorGridHeight += charGridHeight', ''), - Command('K', 'sheet.cursorGridHeight -= charGridHeight', ''), + Command('H', 'sheet.cursorBox.w -= canvasCharWidth', ''), + Command('L', 'sheet.cursorBox.w += canvasCharWidth', ''), + Command('J', 'sheet.cursorBox.h += canvasCharHeight', ''), + Command('K', 'sheet.cursorBox.h -= canvasCharHeight', ''), - Command('zz', 'zoomTo(cursorGridMinX, cursorGridMinY, cursorGridMaxX, cursorGridMaxY)', 'set visible bounds to cursor'), + Command('zz', 'zoomTo(cursorBox)', 'set visible bounds to cursor'), - Command('-', 'tmp=(cursorGridCenterX, cursorGridCenterY); setZoom(zoomlevel*options.zoom_incr); fixPoint(gridCanvasCenterX, gridCanvasCenterY, *tmp)', 'zoom into cursor center'), - Command('+', 'tmp=(cursorGridCenterX, cursorGridCenterY); setZoom(zoomlevel/options.zoom_incr); fixPoint(gridCanvasCenterX, gridCanvasCenterY, *tmp)', 'zoom into cursor center'), - Command('_', 'sheet.gridWidth = 0; sheet.visibleGridWidth = 0; setZoom(1.0); refresh()', 'zoom to fit full extent'), + Command('-', 'tmp=cursorBox.center; setZoom(zoomlevel*options.zoom_incr); fixPoint(plotviewBox.center, tmp)', 'zoom out from cursor center'), + Command('+', 'tmp=cursorBox.center; setZoom(zoomlevel/options.zoom_incr); fixPoint(plotviewBox.center, tmp)', 'zoom into cursor center'), + Command('_', 'sheet.canvasBox = None; sheet.visibleBox = None; setZoom(1.0); refresh()', 'zoom to fit full extent'), + Command('z_', 'sheet.aspectRatio = float(input("aspect ratio=", value=aspectRatio)); refresh()', 'set aspect ratio'), # set cursor box with left click - Command('BUTTON1_PRESSED', 'sheet.cursorGridMinX, sheet.cursorGridMinY = gridMouseX, gridMouseY; sheet.cursorGridWidth=0', 'start cursor box with left mouse button press'), - Command('BUTTON1_RELEASED', 'setCursorSize(gridMouseX, gridMouseY)', 'end cursor box with left mouse button release'), + Command('BUTTON1_PRESSED', 'sheet.cursorBox = Box(*canvasMouse.xy)', 'start cursor box with left mouse button press'), + Command('BUTTON1_RELEASED', 'setCursorSize(canvasMouse)', 'end cursor box with left mouse button release'), - Command('BUTTON3_PRESSED', 'sheet.gridAnchorXY = (gridMouseX, gridMouseY)', 'mark grid point to move'), - Command('BUTTON3_RELEASED', 'fixPoint(canvasMouseX, canvasMouseY, *gridAnchorXY)', 'mark canvas anchor point'), + Command('BUTTON3_PRESSED', 'sheet.anchorPoint = canvasMouse', 'mark grid point to move'), + Command('BUTTON3_RELEASED', 'fixPoint(plotterMouse, anchorPoint)', 'mark canvas anchor point'), - Command('BUTTON4_PRESSED', 'tmp=(gridMouseX,gridMouseY); setZoom(zoomlevel/options.zoom_incr); fixPoint(canvasMouseX, canvasMouseY, *tmp)', 'zoom in with scroll wheel'), - Command('REPORT_MOUSE_POSITION', 'tmp=(gridMouseX,gridMouseY); setZoom(zoomlevel*options.zoom_incr); fixPoint(canvasMouseX, canvasMouseY, *tmp)', 'zoom out with scroll wheel'), + Command('BUTTON4_PRESSED', 'tmp=canvasMouse; setZoom(zoomlevel/options.zoom_incr); fixPoint(plotterMouse, tmp)', 'zoom in with scroll wheel'), + Command('REPORT_MOUSE_POSITION', 'tmp=canvasMouse; setZoom(zoomlevel*options.zoom_incr); fixPoint(plotterMouse, tmp)', 'zoom out with scroll wheel'), - Command('s', 'source.select(list(getRowsInside(*cursorPixelBounds)))', 'select all points within cursor box'), - Command('t', 'source.unselect(list(getRowsInside(*cursorPixelBounds)))', 'toggle selection of all points within cursor box'), - Command('u', 'source.unselect(list(getRowsInside(*cursorPixelBounds)))', 'unselect all points within cursor box'), - Command(ENTER, 'vs=copy(source); vs.rows=list(getRowsInside(*cursorPixelBounds)); vd.push(vs)', ''), + Command('s', 'source.select(list(rowsWithin(plotterCursorBox)))', 'select rows on source sheet contained within canvas cursor'), + Command('t', 'source.toggle(list(rowsWithin(plotterCursorBox)))', 'toggle selection of rows on source sheet contained within canvas cursor'), + Command('u', 'source.unselect(list(rowsWithin(plotterCursorBox)))', 'unselect rows on source sheet contained within canvas cursor'), + Command(ENTER, 'vs=copy(source); vs.rows=list(rowsWithin(plotterCursorBox)); vd.push(vs)', 'Open sheet of source rows contained within canvas cursor'), - Command('gs', 'source.select(list(getRowsInside(*visiblePixelBounds)))', 'select all points visible onscreen'), - Command('gt', 'source.unselect(list(getRowsInside(*visiblePixelBounds)))', 'toggle selection of all points visible onscreen'), - Command('gu', 'source.unselect(list(getRowsInside(*visiblePixelBounds)))', 'unselect all points visible onscreen'), - Command('g'+ENTER, 'vs=copy(source); vs.rows=list(getRowsInside(*visiblePixelBounds)); vd.push(vs)', ''), + Command('gs', 'source.select(list(rowsWithin(plotterVisibleBox)))', 'select rows visible on screen'), + Command('gt', 'source.toggle(list(rowsWithin(plotterVisibleBox)))', 'toggle selection of rows visible on screen'), + Command('gu', 'source.unselect(list(rowsWithin(plotterVisibleBox)))', 'unselect rows visible on screen'), + Command('g'+ENTER, 'vs=copy(source); vs.rows=list(rowsWithin(plotterVisibleBox)); vd.push(vs)', 'open sheet of source rows visible on screen'), ] def __init__(self, name, source=None, **kwargs): super().__init__(name, source=source, **kwargs) - # bounding box of entire grid in grid units, updated when adding point/line/label or recalcBounds - self.gridMinX, self.gridMinY = None, None # derive first bounds on first draw - self.gridWidth, self.gridHeight = None, None - - # bounding box of visible grid, in grid units - self.visibleGridMinX = None - self.visibleGridMinY = None - self.visibleGridWidth = None - self.visibleGridHeight = None - - # bounding box of cursor (should be contained within visible grid?) - self.cursorGridMinX, self.cursorGridMinY = 0, 0 - self.cursorGridWidth, self.cursorGridHeight = None, None + self.canvasBox = None # bounding box of entire canvas, in canvas units + self.visibleBox = None # bounding box of visible canvas, in canvas units + self.cursorBox = None # bounding box of cursor, in canvas units self.zoomlevel = 1.0 self.needsRefresh = False - # bounding box of gridCanvas, in pixels - self.gridpoints = [] # list of (grid_x, grid_y, attr, row) - self.gridlines = [] # list of (grid_x1, grid_y1, grid_x2, grid_y2, attr, row) + self.polylines = [] # list of ([(canvas_x, canvas_y), ...], attr, row) self.gridlabels = [] # list of (grid_x, grid_y, label, attr, row) self.legends = collections.OrderedDict() # txt: attr (visible legends only) @@ -289,18 +330,20 @@ def __init__(self, name, source=None, **kwargs): self.reset() def __len__(self): - return len(self.gridpoints) + len(self.gridlines) + return len(self.polylines) def reset(self): + 'clear everything in preparation for a fresh reload()' + self.polylines.clear() self.legends.clear() self.plotAttrs.clear() - self.unusedAttrs = list(colors[colorname] for colorname in options.plot_colors.split()) + self.unusedAttrs = list(colors[colorname.translate(str.maketrans('_', ' '))] for colorname in options.plot_colors.split())[::-1] def plotColor(self, k): attr = self.plotAttrs.get(k, None) if attr is None: if len(self.unusedAttrs) > 1: - attr = self.unusedAttrs.pop() + attr = self.unusedAttrs.pop(0) legend = ' '.join(str(x) for x in k) else: attr = self.unusedAttrs[0] @@ -313,172 +356,77 @@ def plotColor(self, k): def resetCanvasDimensions(self): super().resetCanvasDimensions() - self.gridCanvasMinX = self.leftMarginPixels - self.gridCanvasMinY = self.topMarginPixels - self.gridCanvasWidth = self.canvasWidth - self.rightMarginPixels - self.leftMarginPixels - self.gridCanvasHeight = self.canvasHeight - self.bottomMarginPixels - self.topMarginPixels + self.plotviewBox = BoundingBox(self.leftMarginPixels, self.topMarginPixels, + self.plotwidth-self.rightMarginPixels, self.plotheight-self.bottomMarginPixels) @property def statusLine(self): - gridstr = 'grid (%s,%s)-(%s,%s)' % (self.gridMinX, self.gridMinY, self.gridMaxX, self.gridMaxY) - vgridstr = 'visibleGrid (%s,%s)-(%s,%s)' % (self.visibleGridMinX, self.visibleGridMinY, self.visibleGridMaxX, self.visibleGridMaxY) - cursorstr = 'cursor (%s,%s)-(%s,%s)' % (self.cursorGridMinX, self.cursorGridMinY, self.cursorGridMaxX, self.cursorGridMaxY) - return ' '.join((gridstr, vgridstr, cursorstr)) + return 'canvas %s visible %s cursor %s' % (self.canvasBox, self.visibleBox, self.cursorBox) @property - def canvasMouseX(self): - return self.mouseX*2 + def canvasMouse(self): + return Point(self.visibleBox.xmin + (self.plotterMouse.x-self.plotviewBox.xmin)/self.xScaler, + self.visibleBox.ymin + (self.plotterMouse.y-self.plotviewBox.ymin)/self.yScaler) - @property - def canvasMouseY(self): - return self.mouseY*4 + def setCursorSize(self, p): + 'sets width based on diagonal corner p' + self.cursorBox = BoundingBox(self.cursorBox.xmin, self.cursorBox.ymin, p.x, p.y) + self.cursorBox.w = max(self.cursorBox.w, self.canvasCharWidth) + self.cursorBox.h = max(self.cursorBox.h, self.canvasCharHeight) @property - def gridMouseX(self): - return self.visibleGridMinX + (self.canvasMouseX-self.gridCanvasMinX)/self.xScaler + def canvasCharWidth(self): + 'Width in canvas units of a single char in the terminal' + return self.visibleBox.w*2/self.plotviewBox.w @property - def gridMouseY(self): - return self.visibleGridMinY + (self.canvasMouseY-self.gridCanvasMinY)/self.yScaler - - def setCursorSize(self, gridX, gridY): - 'sets width based on other side x and y' - if gridX > self.cursorGridMinX: - self.cursorGridWidth = max(gridX - self.cursorGridMinX, self.charGridWidth) - else: - self.cursorGridWidth = max(self.cursorGridMinX - gridX, self.charGridWidth) - self.cursorGridMinX = gridX - - if gridY > self.cursorGridMinY: - self.cursorGridHeight = max(gridY - self.cursorGridMinY, self.charGridHeight) - else: - self.cursorGridHeight = max(self.cursorGridMinY - gridY, self.charGridHeight) - self.cursorGridMinY = gridY + def canvasCharHeight(self): + 'Height in canvas units of a single char in the terminal' + return self.visibleBox.h*4/self.plotviewBox.h @property - def charGridWidth(self): - 'Width in grid units of a single char in the terminal' - return self.visibleGridWidth*2/self.gridCanvasWidth + def plotterVisibleBox(self): + return BoundingBox(self.scaleX(self.visibleBox.xmin), + self.scaleY(self.visibleBox.ymin), + self.scaleX(self.visibleBox.xmax), + self.scaleY(self.visibleBox.ymax)) @property - def charGridHeight(self): - 'Height in grid units of a single char in the terminal' - return self.visibleGridHeight*4/self.gridCanvasHeight - - @property - def gridCanvasCenterX(self): - return self.gridCanvasMinX + self.gridCanvasWidth/2 - - @property - def gridCanvasCenterY(self): - return self.gridCanvasMinY + self.gridCanvasHeight/2 - - @property - def gridMaxX(self): - return self.gridMinX + self.gridWidth - - @property - def gridMaxY(self): - return self.gridMinY + self.gridHeight - - @property - def cursorGridCenterX(self): - return self.cursorGridMinX + self.cursorGridWidth/2 - - @property - def cursorGridCenterY(self): - return self.cursorGridMinY + self.cursorGridHeight/2 - - @property - def cursorGridMaxX(self): - return self.cursorGridMinX + self.cursorGridWidth - - @property - def cursorGridMaxY(self): - return self.cursorGridMinY + self.cursorGridHeight - - @property - def visibleGridMaxX(self): - return self.visibleGridMinX + self.visibleGridWidth - - @property - def visibleGridMaxY(self): - return self.visibleGridMinY + self.visibleGridHeight - - @property - def gridCanvasMaxY(self): - return self.gridCanvasMinY + self.gridCanvasHeight - - @property - def gridCanvasMaxX(self): - return self.gridCanvasMinX + self.gridCanvasWidth - - @property - def cursorGridBounds(self): - return [ self.cursorGridMinX, - self.cursorGridMinY, - self.cursorGridMaxX, - self.cursorGridMaxY - ] - - @property - def visibleGridBounds(self): - return [ self.visibleGridMinX, - self.visibleGridMinY, - self.visibleGridMinX+self.visibleGridWidth, - self.visibleGridMinY+self.visibleGridHeight - ] - - @property - def visiblePixelBounds(self): - return [ self.scaleX(self.visibleGridMinX), - self.scaleY(self.visibleGridMinY), - self.scaleX(self.visibleGridMinX+self.visibleGridWidth), - self.scaleY(self.visibleGridMinY+self.visibleGridHeight) - ] - - @property - def cursorPixelBounds(self): - if self.cursorGridWidth is None: - return [0,0,0,0] - return [ self.scaleX(self.cursorGridMinX), - self.scaleY(self.cursorGridMinY), - self.scaleX(self.cursorGridMinX+self.cursorGridWidth), - self.scaleY(self.cursorGridMinY+self.cursorGridHeight) - ] + def plotterCursorBox(self): + if self.cursorBox is None: + return Box(0,0,0,0) + return BoundingBox(self.scaleX(self.cursorBox.xmin), + self.scaleY(self.cursorBox.ymin), + self.scaleX(self.cursorBox.xmax), + self.scaleY(self.cursorBox.ymax)) def point(self, x, y, attr, row=None): - self.gridpoints.append((x, y, attr, row)) + self.polylines.append(([(x, y)], attr, row)) def line(self, x1, y1, x2, y2, attr, row=None): - self.gridlines.append((x1, y1, x2, y2, attr, row)) - - def polyline(self, vertices, attr, row=None): - 'adds lines for (x,y) vertices of a polygon' - prev_x, prev_y = vertices[0] - for x, y in vertices[1:]: - self.line(prev_x, prev_y, x, y, attr, row) - prev_x, prev_y = x, y - - def polygon(self, vertices, attr, row=None): - 'adds lines for (x,y) vertices of a polygon' - prev_x, prev_y = vertices[-1] - for x, y in vertices: - self.line(prev_x, prev_y, x, y, attr, row) - prev_x, prev_y = x, y + self.polylines.append(([(x1, y1), (x2, y2)], attr, row)) + + def polyline(self, vertexes, attr, row=None): + 'adds lines for (x,y) vertexes of a polygon' + self.polylines.append((vertexes, attr, row)) + + def polygon(self, vertexes, attr, row=None): + 'adds lines for (x,y) vertexes of a polygon' + self.polylines.append((vertexes + [vertexes[0]], attr, row)) def label(self, x, y, text, attr, row=None): self.gridlabels.append((x, y, text, attr, row)) - def fixPoint(self, canvas_x, canvas_y, grid_x, grid_y): - 'adjust visibleGrid so that (grid_x, grid_y) is plotted at (canvas_x, canvas_y)' - self.visibleGridMinX = grid_x - self.gridW(canvas_x-self.gridCanvasMinX) - self.visibleGridMinY = grid_y - self.gridH(canvas_y-self.gridCanvasMinY) + def fixPoint(self, plotterPoint, canvasPoint): + 'adjust visibleBox.xymin so that canvasPoint is plotted at plotterPoint' + self.visibleBox.xmin = canvasPoint.x - self.canvasW(plotterPoint.x-self.plotviewBox.xmin) + self.visibleBox.ymin = canvasPoint.y - self.canvasH(plotterPoint.y-self.plotviewBox.ymin) self.refresh() - def zoomTo(self, x1, y1, x2, y2): - self.fixPoint(self.gridCanvasMinX, self.gridCanvasMinY, x1, y1) - self.zoomlevel=max(self.cursorGridWidth/self.gridWidth, self.cursorGridHeight/self.gridHeight) + def zoomTo(self, bbox): + 'set visible area to bbox, maintaining aspectRatio if applicable' + self.fixPoint(self.plotviewBox.xymin, bbox.xymin) + self.zoomlevel=max(bbox.w/self.canvasBox.w, bbox.h/self.canvasBox.h) def setZoom(self, zoomlevel=None): if zoomlevel: @@ -488,54 +436,35 @@ def setZoom(self, zoomlevel=None): self.plotlegends() def resetBounds(self): - if not self.gridWidth or not self.gridHeight: - xmins = [] - ymins = [] - xmaxs = [] - ymaxs = [] - if self.gridpoints: - xmins.append(min(x for x, y, attr, row in self.gridpoints)) - xmaxs.append(max(x for x, y, attr, row in self.gridpoints)) - ymins.append(min(y for x, y, attr, row in self.gridpoints)) - ymaxs.append(max(y for x, y, attr, row in self.gridpoints)) - if self.gridlines: - xmins.append(min(min(x1, x2) for x1, y1, x2, y2, attr, row in self.gridlines)) - xmaxs.append(max(max(x1, x2) for x1, y1, x2, y2, attr, row in self.gridlines)) - ymins.append(min(min(y1, y2) for x1, y1, x2, y2, attr, row in self.gridlines)) - ymaxs.append(max(max(y1, y2) for x1, y1, x2, y2, attr, row in self.gridlines)) - - if xmins: - self.gridMinX = min(xmins) - self.gridMinY = min(ymins) - self.gridWidth = max(xmaxs) - self.gridMinX - self.gridHeight = max(ymaxs) - self.gridMinY - else: - self.gridWidth = 1.0 - self.gridHeight = 1.0 - - if not self.visibleGridWidth or not self.visibleGridHeight: - # initialize minx/miny, but w/h must be set first - self.visibleGridWidth = self.gridCanvasWidth/self.xScaler - self.visibleGridHeight = self.gridCanvasHeight/self.yScaler - self.visibleGridMinX = self.gridMinX + self.gridWidth/2 - self.visibleGridWidth/2 - self.visibleGridMinY = self.gridMinY + self.gridHeight/2 - self.visibleGridHeight/2 + if not self.canvasBox: + xmin, ymin, xmax, ymax = None, None, None, None + for vertexes, attr, row in self.polylines: + for x, y in vertexes: + if xmin is None or x < xmin: xmin = x + if ymin is None or y < ymin: ymin = y + if xmax is None or x > xmax: xmax = x + if ymax is None or y > ymax: ymax = y + self.canvasBox = BoundingBox(xmin or 0, ymin or 0, xmax or 1, ymax or 1) + + if not self.visibleBox: + # initialize minx/miny, but w/h must be set first to center properly + self.visibleBox = Box(0, 0, self.plotviewBox.w/self.xScaler, self.plotviewBox.h/self.yScaler) + self.visibleBox.xmin = self.canvasBox.xcenter - self.visibleBox.w/2 + self.visibleBox.ymin = self.canvasBox.ycenter - self.visibleBox.h/2 else: - self.visibleGridWidth = self.gridCanvasWidth/self.xScaler - self.visibleGridHeight = self.gridCanvasHeight/self.yScaler + self.visibleBox.w = self.plotviewBox.w/self.xScaler + self.visibleBox.h = self.plotviewBox.h/self.yScaler - if not self.cursorGridWidth or not self.cursorGridHeight: - self.cursorGridMinX = self.visibleGridMinX - self.cursorGridMinY = self.visibleGridMinY - self.cursorGridWidth = self.charGridWidth - self.cursorGridHeight = self.charGridHeight + if not self.cursorBox: + self.cursorBox = Box(self.visibleBox.xmin, self.visibleBox.ymin, self.canvasCharWidth, self.canvasCharHeight) def plotlegends(self): # display labels for i, (legend, attr) in enumerate(self.legends.items()): - self._commands[str(i+1)] = Command(str(i+1), 'togglePixelAttrs(%s)' % attr, '') - if attr in self.disabledAttrs: - attr = colors['238 blue'] - self.plotlegend(i, '%s.%s'%(i+1,legend), attr) + self._commands[str(i+1)] = Command(str(i+1), 'hideAttr(%s, %s not in hiddenAttrs)' % (attr, attr), 'toggle display of "%s"' % legend) + if attr in self.hiddenAttrs: + attr = colors[options.color_graph_hidden] + self.plotlegend(i, '%s:%s'%(i+1,legend), attr) def checkCursor(self): 'override Sheet.checkCursor' @@ -543,75 +472,80 @@ def checkCursor(self): @property def xScaler(self): - xratio = self.gridCanvasWidth/(self.gridWidth*self.zoomlevel) + xratio = self.plotviewBox.w/(self.canvasBox.w*self.zoomlevel) if self.aspectRatio: - yratio = self.gridCanvasHeight/(self.gridHeight*self.zoomlevel) + yratio = self.plotviewBox.h/(self.canvasBox.h*self.zoomlevel) return self.aspectRatio*min(xratio, yratio) else: return xratio @property def yScaler(self): - yratio = self.gridCanvasHeight/(self.gridHeight*self.zoomlevel) + yratio = self.plotviewBox.h/(self.canvasBox.h*self.zoomlevel) if self.aspectRatio: - xratio = self.gridCanvasWidth/(self.gridWidth*self.zoomlevel) - return self.aspectRatio*min(xratio, yratio) + xratio = self.plotviewBox.w/(self.canvasBox.w*self.zoomlevel) + return min(xratio, yratio) else: return yratio def scaleX(self, x): - 'returns canvas x coordinate' - return round(self.gridCanvasMinX+(x-self.visibleGridMinX)*self.xScaler) + 'returns plotter x coordinate' + return round(self.plotviewBox.xmin+(x-self.visibleBox.xmin)*self.xScaler) def scaleY(self, y): - 'returns canvas y coordinate' - return round(self.gridCanvasMinY+(y-self.visibleGridMinY)*self.yScaler) + 'returns plotter y coordinate' + return round(self.plotviewBox.ymin+(y-self.visibleBox.ymin)*self.yScaler) - def gridW(self, canvas_width): - 'canvas X units to grid units' - return canvas_width/self.xScaler + def canvasW(self, plotter_width): + 'plotter X units to canvas units' + return plotter_width/self.xScaler - def gridH(self, canvas_height): - 'canvas Y units to grid units' - return canvas_height/self.yScaler + def canvasH(self, plotter_height): + 'plotter Y units to canvas units' + return plotter_height/self.yScaler def refresh(self): + 'triggers render() on next draw()' self.needsRefresh = True - def plotAll(self): + def render(self): + 'resets plotter, cancels previous render threads, spawns a new render' self.needsRefresh = False cancelThread(*(t for t in self.currentThreads if t.name == 'plotAll_async')) - self.pixels.clear() self.labels.clear() self.resetCanvasDimensions() - self.plotAll_async() + self.render_async() @async - def plotAll_async(self): - 'plots points and lines and text onto the PixelCanvas' + def render_async(self): + 'plots points and lines and text onto the Plotter' self.setZoom() - - xmin, ymin, xmax, ymax = self.visibleGridBounds + bb = self.visibleBox + xmin, ymin, xmax, ymax = bb.xmin, bb.ymin, bb.xmax, bb.ymax xfactor, yfactor = self.xScaler, self.yScaler - gridxmin, gridymin = self.gridCanvasMinX, self.gridCanvasMinY - - for x, y, attr, row in Progress(self.gridpoints): - if ymin <= y and y <= ymax: - if xmin <= x and x <= xmax: - x = gridxmin+(x-xmin)*xfactor - y = gridymin+(y-ymin)*yfactor - self.plotpixel(x, y, attr, row) - - for x1, y1, x2, y2, attr, row in Progress(self.gridlines): - r = self.clipline(x1, y1, x2, y2, xmin, ymin, xmax, ymax) - if r: - x1, y1, x2, y2 = r - x1 = gridxmin+(x1-xmin)*xfactor - y1 = gridymin+(y1-ymin)*yfactor - x2 = gridxmin+(x2-xmin)*xfactor - y2 = gridymin+(y2-ymin)*yfactor - self.plotline(x1, y1, x2, y2, attr, row) + plotxmin, plotymin = self.plotviewBox.xmin, self.plotviewBox.ymin + + for vertexes, attr, row in Progress(self.polylines): + if len(vertexes) == 1: # single point + x1, y1 = vertexes[0] + if xmin <= x1 <= xmax and ymin <= y1 <= ymax: + x = plotxmin+(x1-xmin)*xfactor + y = plotymin+(y1-ymin)*yfactor + self.plotpixel(round(x), round(y), attr, row) + continue + + prev_x, prev_y = vertexes[0] + for x, y in vertexes[1:]: + r = clipline(prev_x, prev_y, x, y, xmin, ymin, xmax, ymax) + if r: + x1, y1, x2, y2 = r + x1 = plotxmin+(x1-xmin)*xfactor + y1 = plotymin+(y1-ymin)*yfactor + x2 = plotxmin+(x2-xmin)*xfactor + y2 = plotymin+(y2-ymin)*yfactor + self.plotline(x1, y1, x2, y2, attr, row) + prev_x, prev_y = x, y for x, y, text, attr, row in Progress(self.gridlabels): self.plotlabel(self.scaleX(x), self.scaleY(y), text, attr, row) diff --git a/visidata/clipboard.py b/visidata/clipboard.py index 9f542a105..c65922a5d 100644 --- a/visidata/clipboard.py +++ b/visidata/clipboard.py @@ -12,7 +12,7 @@ globalCommand('gd', 'vd.cliprows = list((None, i, r) for i, r in enumerate(selectedRows)); deleteSelected()', 'delete all selected rows and move them to clipboard') globalCommand('gy', 'vd.cliprows = list((None, i, r) for i, r in enumerate(selectedRows))', 'copy all selected rows to clipboard') -globalCommand('zy', 'vd.clipvalue = cursorDisplay', 'copy this cell to clipboard') +globalCommand('zy', 'vd.clipvalue = cursorDisplay', 'copy current cell to clipboard') globalCommand('gzp', 'cursorCol.setValues(selectedRows or rows, vd.clipvalue)', 'set contents of current column for selected rows to last clipboard value') globalCommand('zp', 'cursorCol.setValue(cursorRow, vd.clipvalue)', 'set contents of current column for current row to last clipboard value') diff --git a/visidata/cmdlog.py b/visidata/cmdlog.py index 7725d8e7d..4a858020e 100644 --- a/visidata/cmdlog.py +++ b/visidata/cmdlog.py @@ -9,11 +9,11 @@ option('disp_replay_pause', '‖', 'status indicator for paused replay') option('replay_movement', False, 'insert movements during replay') -globalCommand('D', 'vd.push(vd.cmdlog)', 'open CommandLog') -globalCommand('^D', 'saveSheet(vd.cmdlog, input("save to: ", "filename", value=fnSuffix("cmdlog-{0}.vd") or "cmdlog.vd"))', 'save CommandLog to new .vd file') -globalCommand('^U', 'CommandLog.togglePause()', 'pause/resume replay') -globalCommand(' ', 'CommandLog.currentReplay.advance()', 'execute next row in replaying sheet') -globalCommand('^K', 'CommandLog.currentReplay.cancel()', 'cancel current replay') +globalCommand('D', 'vd.push(vd.cmdlog)', 'open CommandLog', 'open-cmdlog') +globalCommand('^D', 'saveSheet(vd.cmdlog, inputFilename("save to: ", value=fnSuffix("cmdlog-{0}.vd") or "cmdlog.vd"))', 'save CommandLog to new .vd file', 'save-cmdlog') +globalCommand('^U', 'CommandLog.togglePause()', 'pause/resume replay', 'toggle-replay') +globalCommand('^I', '(CommandLog.currentReplay or error("no replay to advance")).advance()', 'execute next row in replaying sheet', 'step-replay') +globalCommand('^K', '(CommandLog.currentReplay or error("no replay to cancel")).cancel()', 'cancel current replay', 'cancel-replay') #globalCommand('KEY_BACKSPACE', 'vd.cmdlog.undo()', 'remove last action on commandlog and replay') @@ -21,11 +21,12 @@ globalCommand('status', 'status(input("status: ", display=False))', 'show given status message') # not necessary to log movements and scrollers -nonLogKeys = 'KEY_DOWN KEY_UP KEY_NPAGE KEY_PPAGE kDOWN kUP j k gj gk ^F ^B r < > { } / ? n N g/ g?'.split() +nonLogKeys = 'KEY_DOWN KEY_UP KEY_NPAGE KEY_PPAGE kDOWN kUP j k gj gk ^F ^B r < > { } / ? n N gg G g/ g?'.split() nonLogKeys += 'KEY_LEFT KEY_RIGHT h l gh gl c'.split() nonLogKeys += 'zk zj zt zz zb zh zl zKEY_LEFT zKEY_RIGHT'.split() -nonLogKeys += '^L ^C ^U ^D KEY_RESIZE'.split() +nonLogKeys += '^L ^C ^U ^K ^I ^D KEY_RESIZE'.split() +option('rowkey_prefix', 'キ', 'string prefix for rowkey in the cmdlog') def itemsetter(i): def g(obj, v): @@ -60,9 +61,12 @@ def getColVisibleIdxByFullName(sheet, name): if name == c.name: return i -def getRowIdxByKey(sheet, keyvals): +def keystr(k): + return ','.join(map(str, k)) + +def getRowIdxByKey(sheet, k): for i, r in enumerate(sheet.rows): - if sheet.keyvals(r) == keyvals: + if keystr(sheet.rowkey(r)) == k: return i def loggable(keystrokes): @@ -126,13 +130,20 @@ def beforeExecHook(self, sheet, keystrokes, args=''): return # don't record editlog commands if self.currentActiveRow: self.afterExecSheet(sheet, False, '') - if keystrokes == 'o': - sheetname, colname, rowname = '', '', '' - else: + + cmd = sheet.getCommand(keystrokes) + sheetname, colname, rowname = '', '', '' + if sheet and keystrokes != 'o': + contains = lambda s, *substrs: any((a in s) for a in substrs) sheetname = sheet.name - colname = sheet.cursorCol.name or sheet.visibleCols.index(sheet.cursorCol) - rowname = sheet.cursorRowIndex - self.currentActiveRow = CommandLogRow([sheetname, colname, rowname, keystrokes, args, sheet.getCommand(keystrokes).helpstr]) + if sheet.rows and contains(cmd.execstr, 'cursorValue', 'cursorCell', 'cursorRow'): + k = sheet.rowkey(sheet.cursorRow) + rowname = (options.rowkey_prefix + keystr(k)) if k else sheet.cursorRowIndex + + if contains(cmd.execstr, 'cursorValue', 'cursorCell', 'cursorCol', 'cursorVisibleCol'): + colname = sheet.cursorCol.name or sheet.visibleCols.index(sheet.cursorCol) + + self.currentActiveRow = CommandLogRow([sheetname, colname, rowname, keystrokes, args, cmd.helpstr]) def afterExecSheet(self, sheet, escaped, err): 'Records currentActiveRow' @@ -165,7 +176,7 @@ def getSheet(self, sheetname): @classmethod def togglePause(self): if not CommandLog.currentReplay: - status('no replay in progress') + status('no replay to pause') else: if self.paused: CommandLog.currentReplay.advance() @@ -192,7 +203,8 @@ def moveToReplayContext(self, r): try: rowidx = int(r.row) except ValueError: - rowidx = getRowIdxByKey(vs, r.row) + k = r.row[1:] # trim rowkey_prefix + rowidx = getRowIdxByKey(vs, k) if rowidx is None: error('no row %s' % r.row) diff --git a/visidata/data.py b/visidata/data.py index cf3539590..6f50f94c0 100644 --- a/visidata/data.py +++ b/visidata/data.py @@ -27,7 +27,7 @@ globalCommand('a', 'rows.insert(cursorRowIndex+1, newRow()); cursorDown(1)', 'append a blank row') globalCommand('ga', 'for r in range(int(input("add rows: "))): addRow(newRow())', 'add N blank rows') -globalCommand('f', 'fillNullValues(cursorCol, selectedRows or rows)', 'fill null cells in current column with previous non-null value') +globalCommand('f', 'fillNullValues(cursorCol, selectedRows or rows)', 'fills null cells in current column with contents of non-null cells up the current column') def fillNullValues(col, rows): 'Fill null cells in col with the previous non-null value' @@ -51,19 +51,19 @@ def updateColNames(sheet): if not c._name: c.name = "_".join(c.getDisplayValue(r) for r in sheet.selectedRows or [sheet.cursorRow]) -globalCommand('z^', 'sheet.cursorCol.name = cursorDisplay', 'set current column name to value in current cell') -globalCommand('g^', 'updateColNames(sheet)', 'set visible column names to values in selected rows (or current row)') -globalCommand('gz^', 'sheet.cursorCol.name = "_".join(sheet.cursorCol.getDisplayValue(r) for r in selectedRows or [cursorRow]) ', 'set current column name to combined values in selected rows (or current row)') +globalCommand('z^', 'sheet.cursorCol.name = cursorDisplay', 'set name of current column to contents of current cell') +globalCommand('g^', 'updateColNames(sheet)', 'set names of all visible columns to contents of selected rows (or current row)') +globalCommand('gz^', 'sheet.cursorCol.name = "_".join(sheet.cursorCol.getDisplayValue(r) for r in selectedRows or [cursorRow]) ', 'set current column name to combined contents of current cell in selected rows (or current row)') # gz^ with no selectedRows is same as z^ -globalCommand('o', 'vd.push(openSource(input("open: ", "filename")))', 'open input in VisiData') -globalCommand('^S', 'saveSheet(sheet, input("save to: ", "filename", value=getDefaultSaveName(sheet)), options.confirm_overwrite)', 'save current sheet to filename in format determined by extension (default .tsv)') +globalCommand('o', 'vd.push(openSource(inputFilename("open: ")))', 'open input in VisiData') +globalCommand('^S', 'saveSheet(sheet, inputFilename("save to: ", value=getDefaultSaveName(sheet)), options.confirm_overwrite)', 'save current sheet to filename in format determined by extension (default .tsv)') -globalCommand('z=', 'status(evalexpr(input("status=", "expr"), cursorRow))', 'evaluate Python expression on current row and display result on status line') +globalCommand('z=', 'cursorCol.setValue(cursorRow, evalexpr(inputExpr("set cell=")))', 'set current cell to result of evaluated Python expression on current row') -globalCommand('gz=', 'for r, v in zip(selectedRows or rows, eval(input("set column= ", "expr"))): cursorCol.setValue(r, v)', 'set selected rows in this column to the values in the given Python sequence expression') +globalCommand('gz=', 'for r, v in zip(selectedRows or rows, eval(input("set column= ", "expr", completer=CompleteExpr()))): cursorCol.setValue(r, v)', 'set current column for selected rows to the items in result of Python sequence expression') -globalCommand('A', 'vd.push(newSheet(int(input("num columns for new sheet: "))))', 'open new blank sheet with number columns') +globalCommand('A', 'vd.push(newSheet(int(input("num columns for new sheet: "))))', 'open new blank sheet with N columns') globalCommand('gKEY_F(1)', 'help-commands') # vdtui generic commands sheet @@ -74,7 +74,6 @@ def updateColNames(sheet): globalCommand('KEY_F(1)', 'z?') def openManPage(): - import subprocess from pkg_resources import resource_filename with SuspendCurses(): os.system(' '.join(['man', resource_filename(__name__, 'man/vd.1')])) @@ -82,6 +81,29 @@ def openManPage(): def newSheet(ncols): return Sheet('unnamed', columns=[ColumnItem('', i, width=8) for i in range(ncols)]) +def inputFilename(prompt, *args, **kwargs): + return input(prompt, "filename", *args, completer=completeFilename, **kwargs) + +def completeFilename(val, state): + i = val.rfind('/') + if i < 0: # no / + base = '' + partial = val + elif i == 0: # root / + base = '/' + partial = val[1:] + else: + base = val[:i] + partial = val[i+1:] + + files = [] + for f in os.listdir(Path(base or '.').resolve()): + if f.startswith(partial): + files.append(os.path.join(base, f)) + + files.sort() + return files[state%len(files)] + def getDefaultSaveName(sheet): src = getattr(sheet, 'source', None) if isinstance(src, Path): @@ -121,15 +143,15 @@ def reload(self): def openSource(p, filetype=None): - 'calls open_ext(Path) or openurl_scheme(UrlPath)' + 'calls open_ext(Path) or openurl_scheme(UrlPath, filetype)' if isinstance(p, str): if '://' in p: - p = UrlPath(p) - filetype = filetype or p.scheme - openfunc = 'openurl_' + p.scheme - vs = getGlobals()[openfunc](p) + return openSource(UrlPath(p), filetype) # convert to Path and recurse else: return openSource(Path(p), filetype) # convert to Path and recurse + elif isinstance(p, UrlPath): + openfunc = 'openurl_' + p.scheme + return getGlobals()[openfunc](p, filetype=filetype) elif isinstance(p, Path): if not filetype: filetype = options.filetype or p.suffix @@ -178,7 +200,7 @@ def _getTsvHeaders(fp, nlines): i = 0 while i < nlines: L = next(fp) - L = L[:-1] + L = L.rstrip('\n') if L: headers.append(L.split(options.delimiter)) i += 1 @@ -227,7 +249,7 @@ def reload_tsv_sync(vs, **kwargs): L = next(fp) except StopIteration: break - L = L[:-1] + L = L.rstrip('\n') if L: vs.addRow(L.split(delim)) prog.addProgress(len(L)) diff --git a/visidata/describe.py b/visidata/describe.py index bf8ea9b52..1160a803d 100644 --- a/visidata/describe.py +++ b/visidata/describe.py @@ -42,17 +42,18 @@ class DescribeSheet(Sheet): DescribeColumn('mean', type=float), DescribeColumn('stdev', type=float), ] - commands = ColumnsSheet.commands + [ + commands = [ Command('zs', 'source.select(cursorValue)', 'select rows on source sheet which are being described in current cell'), Command('zu', 'source.unselect(cursorValue)', 'unselect rows on source sheet which are being described in current cell'), Command('z'+ENTER, 'vs=copy(source); vs.rows=cursorValue; vs.name+="_%s_%s"%(cursorRow.name,cursorCol.name); vd.push(vs)', 'open copy of source sheet with rows described in current cell'), Command(ENTER, 'vd.push(SheetFreqTable(source, cursorRow))', 'open a Frequency Table sheet grouped on column referenced in current row'), - Command('!', 'source.toggleKeyColumn(source.columns.index(cursorRow))', 'toggle key column on source sheet') + Command('!', 'source.toggleKeyColumn(source.columns.index(cursorRow))', 'pin current column on left as a key column on source sheet') ] colorizers = [ Colorizer('row', 7, lambda self,c,r,v: options.color_key_col if r in self.source.keyCols else None), ] + @async def reload(self): self.rows = list(self.source.columns) # column deleting/reordering here does not affect actual columns self.describeData = { col: {} for col in self.source.columns } @@ -69,7 +70,7 @@ def reloadColumn(self, srccol): d['errors'] = list() d['nulls'] = list() d['distinct'] = set() - for sr in self.sourceRows: + for sr in Progress(self.sourceRows): try: v = srccol.getValue(sr) if isNull(v): @@ -83,7 +84,7 @@ def reloadColumn(self, srccol): d['mode'] = self.calcStatistic(d, mode, vals) if isNumeric(srccol): - for func in [mode, min, max, median, mean, stdev]: + for func in [min, max, median, mean, stdev]: d[func.__name__] = self.calcStatistic(d, func, vals) def calcStatistic(self, d, func, *args, **kwargs): diff --git a/visidata/freeze.py b/visidata/freeze.py index 7337fdc8b..204429360 100644 --- a/visidata/freeze.py +++ b/visidata/freeze.py @@ -3,14 +3,28 @@ globalCommand("'", 'addColumn(StaticColumn(sheet.rows, cursorCol), cursorColIndex+1)', 'add a frozen copy of current column with all cells evaluated') globalCommand("g'", 'vd.push(StaticSheet(sheet)); status("pushed frozen copy of "+name)', 'open a frozen copy of current sheet with all visible columns evaluated') -globalCommand("z'", 'cursorCol._cachedValues = collections.OrderedDict(); status("added cache to " + cursorCol.name)', 'add/reset cache for this column') -globalCommand("gz'", 'for c in visibleCols: c._cachedValues = collections.OrderedDict()', 'add/reset cache for all visible columns') +globalCommand("z'", 'resetCache(cursorCol)', 'add/reset cache for current column') +globalCommand("gz'", 'resetCache(*visibleCols)', 'add/reset cache for all visible columns') globalCommand("zg'", "gz'") +def resetCache(self, *cols): + for col in cols: + col._cachedValues = collections.OrderedDict() + status("reset cache for " + (cols[0].name if len(cols) == 1 else str(len(cols))+" columns")) +Sheet.resetCache = resetCache def StaticColumn(rows, col): c = deepcopy(col) - frozenData = {id(r):col.getValue(r) for r in rows} + frozenData = {} + @async + def _calcRows(sheet): + for r in Progress(rows): + try: + frozenData[id(r)] = col.getValue(r) + except Exception as e: + frozenData[id(r)] = e + + _calcRows(col.sheet) c.calcValue=lambda row,d=frozenData: d[id(row)] c.setter=lambda col,row,val,d=frozenData: setitem(d, id(row), val) c.name = c.name + '_frozen' diff --git a/visidata/freqtbl.py b/visidata/freqtbl.py index 79a4ab91a..175cd162b 100644 --- a/visidata/freqtbl.py +++ b/visidata/freqtbl.py @@ -3,7 +3,7 @@ from visidata import * globalCommand('F', 'vd.push(SheetFreqTable(sheet, cursorCol))', 'open Frequency Table grouped on current column') globalCommand('gF', 'vd.push(SheetFreqTable(sheet, *keyCols))', 'open Frequency Table grouped by all key columns on the source sheet') -globalCommand('zF', 'vd.push(SheetFreqTable(sheet, Column("Total", getter=lambda col,row: "Total")))', 'open a one-line summary for all selected rows') +globalCommand('zF', 'vd.push(SheetFreqTable(sheet, Column("Total", getter=lambda col,row: "Total")))', 'open one-line summary for all selected rows') theme('disp_histogram', '*', 'histogram element character') option('disp_histolen', 80, 'width of histogram column') @@ -34,8 +34,8 @@ class SheetFreqTable(Sheet): Command('s', 'select([cursorRow]); cursorDown(1)', 'select these entries in source sheet'), Command('u', 'unselect([cursorRow]); cursorDown(1)', 'unselect these entries in source sheet'), - Command(ENTER, 'vs = copy(source); vs.name += "_"+valueNames(cursorRow[0]); vs.rows=copy(cursorRow[1]); vd.push(vs)', 'push new sheet with only source rows for this value'), -# Command('w', 'options.histogram_even_interval = not options.histogram_even_interval; reload()', 'toggle histogram_even_interval option') + Command(ENTER, 'vs = copy(source); vs.name += "_"+valueNames(cursorRow[0]); vs.rows=copy(cursorRow[1]); vd.push(vs)', 'open sheet of source rows which are grouped in current cell'), +# Command('v', 'options.histogram_even_interval = not options.histogram_even_interval; reload()', 'toggle histogram_even_interval option') ] def __init__(self, sheet, *columns): diff --git a/visidata/graph.py b/visidata/graph.py index f4423bfbd..7cea91251 100644 --- a/visidata/graph.py +++ b/visidata/graph.py @@ -2,8 +2,8 @@ option('color_graph_axis', 'bold', 'color for graph axis labels') -globalCommand('.', 'vd.push(GraphSheet(sheet.name+"_graph", sheet, selectedRows or rows, keyCols, [cursorCol]))', 'graph the current column vs the first key column (or row number)') -globalCommand('g.', 'vd.push(GraphSheet(sheet.name+"_graph", sheet, selectedRows or rows, keyCols, numericCols(nonKeyVisibleCols)))', 'graph all numeric columns vs the first key column (or row number)') +globalCommand('.', 'vd.push(GraphSheet(sheet.name+"_graph", sheet, selectedRows or rows, keyCols, [cursorCol]))', 'graph the current column vs key columns Numeric key column is on the x-axis, while categorical key columns determin color') +globalCommand('g.', 'vd.push(GraphSheet(sheet.name+"_graph", sheet, selectedRows or rows, keyCols, numericCols(nonKeyVisibleCols)))', 'open a graph of all visible numeric columns vs key column') def numericCols(cols): @@ -11,64 +11,44 @@ def numericCols(cols): return [c for c in cols if isNumeric(c)] -class InvertedYGridCanvas(GridCanvas): - commands = GridCanvas.commands + [ +class InvertedCanvas(Canvas): + commands = Canvas.commands + [ # swap directions of up/down - Command('move-up', 'sheet.cursorGridMinY += cursorGridHeight', 'move cursor up'), - Command('move-down', 'sheet.cursorGridMinY -= cursorGridHeight', 'move cursor down'), + Command('move-up', 'sheet.cursorBox.ymin += cursorBox.h', 'move cursor up'), + Command('move-down', 'sheet.cursorBox.ymin -= cursorBox.h', 'move cursor down'), - Command('zj', 'sheet.cursorGridMinY -= charGridHeight', 'move cursor down one line'), - Command('zk', 'sheet.cursorGridMinY += charGridHeight', 'move cursor up one line'), + Command('zj', 'sheet.cursorBox.ymin -= canvasCharHeight', 'move cursor down one line'), + Command('zk', 'sheet.cursorBox.ymin += canvasCharHeight', 'move cursor up one line'), - Command('J', 'sheet.cursorGridHeight -= charGridHeight', 'decrease cursor height'), - Command('K', 'sheet.cursorGridHeight += charGridHeight', 'increase cursor height'), - - Command('zz', 'zoomTo(cursorGridMinX, cursorGridMinY, cursorGridMaxX, cursorGridMaxY)', 'set visible bounds to cursor'), + Command('J', 'sheet.cursorBox.h -= canvasCharHeight', 'decrease cursor height'), + Command('K', 'sheet.cursorBox.h += canvasCharHeight', 'increase cursor height'), ] - def zoomTo(self, x1, y1, x2, y2): - self.fixPoint(self.gridCanvasMinX, self.gridCanvasMaxY, x1, y1) - self.zoomlevel=max(self.cursorGridWidth/self.gridWidth, self.cursorGridHeight/self.gridHeight) + def zoomTo(self, bbox): + super().zoomTo(bbox) + self.fixPoint(Point(self.plotviewBox.xmin, self.plotviewBox.ymax), bbox.xymin) def plotpixel(self, x, y, attr, row=None): - y = self.gridCanvasMaxY-y+4 - self.pixels[round(y)][round(x)][attr].append(row) - - def scaleY(self, grid_y): - 'returns canvas y coordinate, with y-axis inverted' - canvas_y = super().scaleY(grid_y) - return (self.gridCanvasMaxY-canvas_y+4) - - def gridY(self, canvas_y): - return (self.gridCanvasMaxY-canvas_y)/self.yScaler - - def fixPoint(self, canvas_x, canvas_y, grid_x, grid_y): - 'adjust visibleGrid so that (grid_x, grid_y) is plotted at (canvas_x, canvas_y)' - self.visibleGridMinX = grid_x - self.gridW(canvas_x-self.gridCanvasMinX) - self.visibleGridMinY = grid_y - self.gridH(self.gridCanvasMaxY-canvas_y) - self.refresh() + y = self.plotviewBox.ymax-y+4 + self.pixels[y][x][attr].append(row) - @property - def gridMouseY(self): - return self.visibleGridMinY + (self.gridCanvasMaxY-self.canvasMouseY)/self.yScaler + def scaleY(self, canvasY): + 'returns plotter y coordinate, with y-axis inverted' + plotterY = super().scaleY(canvasY) + return (self.plotviewBox.ymax-plotterY+4) - @property - def cursorPixelBounds(self): - x1, y1, x2, y2 = super().cursorPixelBounds - return x1, y2, x2, y1 # reverse top/bottom + def canvasH(self, plotterY): + return (self.plotviewBox.ymax-plotterY)/self.yScaler @property - def visiblePixelBounds(self): - 'invert y-axis' - return [ self.scaleX(self.visibleGridMinX), - self.scaleY(self.visibleGridMaxY), - self.scaleX(self.visibleGridMaxX), - self.scaleY(self.visibleGridMinY), - ] + def canvasMouse(self): + p = super().canvasMouse + p.y = self.visibleBox.ymin + (self.plotviewBox.ymax-self.plotterMouse.y)/self.yScaler + return p # provides axis labels, legend -class GraphSheet(InvertedYGridCanvas): +class GraphSheet(InvertedCanvas): def __init__(self, name, sheet, rows, xcols, ycols, **kwargs): super().__init__(name, sheet, sourceRows=rows, **kwargs) @@ -80,21 +60,21 @@ def reload(self): nerrors = 0 nplotted = 0 - self.gridpoints.clear() self.reset() status('loading data points') catcols = [c for c in self.xcols if not isNumeric(c)] - numcol = numericCols(self.xcols)[0] + numcols = numericCols(self.xcols) for ycol in self.ycols: for rownum, row in enumerate(Progress(self.sourceRows)): # rows being plotted from source try: k = tuple(c.getValue(row) for c in catcols) if catcols else (ycol.name,) - attr = self.plotColor(k) - graph_x = float(numcol.getTypedValue(row)) if self.xcols else rownum - graph_y = ycol.getTypedValue(row) + # convert deliberately to float (to e.g. linearize date) + graph_x = float(numcols[0].type(numcols[0].getValue(row))) if numcols else rownum + graph_y = ycol.type(ycol.getValue(row)) + attr = self.plotColor(k) self.point(graph_x, graph_y, attr, row) nplotted += 1 except Exception: @@ -113,28 +93,25 @@ def setZoom(self, zoomlevel=None): self.createLabels() def add_y_axis_label(self, frac): - amt = self.visibleGridMinY + frac*(self.visibleGridHeight) - if isinstance(self.gridMinY, int): + amt = self.visibleBox.ymin + frac*self.visibleBox.h + if isinstance(self.canvasBox.ymin, int): txt = '%d' % amt - elif isinstance(self.gridMinY, float): + elif isinstance(self.canvasBox.ymin, float): txt = '%.02f' % amt else: txt = str(frac) - # plot y-axis labels on the far left of the canvas, but within the gridCanvas height-wise + # plot y-axis labels on the far left of the canvas, but within the plotview height-wise attr = colors[options.color_graph_axis] - self.plotlabel(0, self.gridCanvasMinY + (1.0-frac)*self.gridCanvasHeight, txt, attr) + self.plotlabel(0, self.plotviewBox.ymin + (1.0-frac)*self.plotviewBox.h, txt, attr) def add_x_axis_label(self, frac): - amt = self.visibleGridMinX + frac*self.visibleGridWidth + amt = self.visibleBox.xmin + frac*self.visibleBox.w txt = ','.join(xcol.format(xcol.type(amt)) for xcol in self.xcols if isNumeric(xcol)) - # plot x-axis labels below the gridCanvasMaxY, but within the gridCanvas width-wise + # plot x-axis labels below the plotviewBox.ymax, but within the plotview width-wise attr = colors[options.color_graph_axis] - self.plotlabel(self.gridCanvasMinX+frac*self.gridCanvasWidth, self.gridCanvasMaxY+4, txt, attr) - - def plotAll(self): - super().plotAll() + self.plotlabel(self.plotviewBox.xmin+frac*self.plotviewBox.w, self.plotviewBox.ymax+4, txt, attr) def createLabels(self): self.gridlabels = [] @@ -153,8 +130,8 @@ def createLabels(self): self.add_x_axis_label(0.25) self.add_x_axis_label(0.00) - # TODO: if 0 line is within visibleGrid, explicitly draw on the axis + # TODO: if 0 line is within visible bounds, explicitly draw the axis # TODO: grid lines corresponding to axis labels xname = ','.join(xcol.name for xcol in self.xcols if isNumeric(xcol)) or 'row#' - self.plotlabel(0, self.gridCanvasMaxY+4, '%*s»' % (int(self.leftMarginPixels/2-2), xname), colors[options.color_graph_axis]) + self.plotlabel(0, self.plotviewBox.ymax+4, '%*s»' % (int(self.leftMarginPixels/2-2), xname), colors[options.color_graph_axis]) diff --git a/visidata/loaders/html.py b/visidata/loaders/html.py new file mode 100644 index 000000000..1964bfe3b --- /dev/null +++ b/visidata/loaders/html.py @@ -0,0 +1,72 @@ +import html +from visidata import * + + +def open_html(p): + return HtmlTablesSheet(p.name, source=p) + +# rowdef: lxml.html.HtmlElement +class HtmlTablesSheet(Sheet): + rowtype = 'tables' + columns = [ + Column('tag', getter=lambda col,row: row.tag), + Column('id', getter=lambda col,row: row.attrib.get('id')), + Column('classes', getter=lambda col,row: row.attrib.get('class')), + ] + commands = [ Command(ENTER, 'vd.push(HtmlTableSheet(name=cursorRow.attrib.get("id", "table_" + str(cursorRowIndex)), source=cursorRow))', 'open this table') ] + @async + def reload(self): + import lxml.html + from lxml import etree + utf8_parser = etree.HTMLParser(encoding='utf-8') + with self.source.open_text() as fp: + html = lxml.html.etree.parse(fp, parser=utf8_parser) + self.rows = [] + for e in html.iter(): + if e.tag == 'table': + self.addRow(e) + +class HtmlTableSheet(Sheet): + rowtype = 'rows' + columns = [] + + @async + def reload(self): + self.rows = [] + for r in self.source.iter(): + if r.tag == 'tr': + row = [(c.text or '').strip() for c in r.getchildren()] + if any(row): + self.addRow(row) + for i, name in enumerate(self.rows[0]): + self.addColumn(ColumnItem(name, i)) + self.rows = self.rows[1:] + +@async +def save_html(sheet, fn): + 'Save sheet as
in an HTML file.' + + with open(fn, 'w', encoding='ascii', errors='xmlcharrefreplace') as fp: + fp.write('\n'.format(sheetname=sheet.name)) + + # headers + fp.write('') + for col in sheet.visibleCols: + contents = html.escape(col.name) + fp.write(''.format(colname=contents)) + fp.write('\n') + + # rows + for r in Progress(sheet.rows): + fp.write('') + for col in sheet.visibleCols: + fp.write('') + fp.write('\n') + + fp.write('
{colname}
') + fp.write(html.escape(col.getDisplayValue(r))) + fp.write('
') + status('%s save finished' % fn) + + + diff --git a/visidata/loaders/http.py b/visidata/loaders/http.py new file mode 100644 index 000000000..940338758 --- /dev/null +++ b/visidata/loaders/http.py @@ -0,0 +1,16 @@ +from visidata import * + +class HttpPath(PathFd): + def __init__(self, url, req): + from urllib.parse import urlparse + obj = urlparse(url) + super().__init__(obj.path, req) + self.req = req + + +def openurl_http(p, filetype=None): + import requests + r = requests.get(p.url, stream=True) + return openSource(HttpPath(p.url, r.iter_lines(decode_unicode=True)), filetype=filetype) + +openurl_https = openurl_http diff --git a/visidata/loaders/json.py b/visidata/loaders/json.py new file mode 100644 index 000000000..ec09fc9f6 --- /dev/null +++ b/visidata/loaders/json.py @@ -0,0 +1,62 @@ +import json + +from visidata import * + + +def open_json(p): + return JSONSheet(p.name, source=p, jsonlines=False) + +def open_jsonl(p): + return JSONSheet(p.name, source=p, jsonlines=True) + + +class JSONSheet(Sheet): + commands = [Command(ENTER, 'pyobj-dive')] + @async + def reload(self): + if not self.jsonlines: + try: + self.reload_json() + except json.decoder.JSONDecodeError: + status('trying jsonl') + self.jsonlines = True + + if self.jsonlines: + self.reload_jsonl() + + def reload_json(self): + self.rows = [] + with self.source.open_text() as fp: + r = json.load(fp, object_hook=self.addRow) + self.rows = [r] if isinstance(r, dict) else r + self.columns = DictKeyColumns(self.rows[0]) + self.recalc() + + def reload_jsonl(self): + with self.source.open_text() as fp: + self.rows = [] + for L in fp: + self.addRow(json.loads(L)) + + def addRow(self, row): + if not self.rows: + self.columns = DictKeyColumns(row) + self.recalc() + self.rows.append(row) + return row + +@async +def save_json(vs, fn): + def rowdict(cols, row): + d = {} + for col in cols: + try: + d[col.name] = col.getValue(row) + except Exception: + pass + return d + + with open(fn, 'w') as fp: + vcols = vs.visibleCols + for chunk in json.JSONEncoder().iterencode([rowdict(vcols, r) for r in Progress(vs.rows)]): + fp.write(chunk) diff --git a/visidata/loaders/mbtiles.py b/visidata/loaders/mbtiles.py index 99720dd21..b8c9fd33d 100644 --- a/visidata/loaders/mbtiles.py +++ b/visidata/loaders/mbtiles.py @@ -17,17 +17,6 @@ def getListDepth(L): return 0 return getListDepth(L[0]) + 1 -def getTile(con, zoom_level, tile_col, tile_row): - import mapbox_vector_tile - - tile_data = con.execute(''' - SELECT tile_data FROM tiles - WHERE zoom_level = ? - AND tile_column = ? - AND tile_row = ?''', (zoom_level, tile_col, tile_row)).fetchone()[0] - - return mapbox_vector_tile.decode(gzip.decompress(tile_data)) - def getFeatures(tile_data): for layername, layer in tile_data.items(): for feat in layer['features']: @@ -42,31 +31,45 @@ class MbtilesSheet(Sheet): ] commands = [ - Command(ENTER, 'vd.push(PbfSheet(tilename(cursorRow)+"_foo", tile_data=getTile(con, *cursorRow)))', 'view this tile'), - Command('.', 'vd.push(PbfCanvas(tilename(cursorRow)+"_map", source=PbfSheet("foo"), sourceRows=list(getFeatures(getTile(con, *cursorRow)))))', 'view this tile'), - Command('g.', 'vd.push(PbfCanvas(tilename(cursorRow)+"_map", source=PbfSheet("foo"), sourceRows=sum((list(getFeatures(getTile(sheet.con, *r))) for r in selectedRows or rows), [])))', 'view selected tiles'), -# Command('1', 'vd.push(load_pyobj("foo", getTile(con, *cursorRow)))', 'push raw data for this tile'), + Command(ENTER, 'vd.push(PbfSheet(tilename(cursorRow), source=sheet, sourceRow=cursorRow))', 'open this tile'), + Command('.', 'tn=tilename(cursorRow); vd.push(PbfCanvas(tn+"_map", source=PbfSheet(tn, sourceRows=list(getFeatures(getTile(*cursorRow)))))', 'plot this tile'), +# Command('g.', 'tn=tilename(cursorRow); vd.push(PbfCanvas(tn+"_map", source=PbfSheet(tn), sourceRows=sum((list(getFeatures(getTile(*r))) for r in selectedRows or rows), [])))', 'plot selected tiles'), ] def tilename(self, row): return ",".join(str(x) for x in row) + def getTile(self, zoom_level, tile_col, tile_row): + import mapbox_vector_tile + + con = sqlite3.connect(self.source.resolve()) + tile_data = con.execute(''' + SELECT tile_data FROM tiles + WHERE zoom_level = ? + AND tile_column = ? + AND tile_row = ?''', (zoom_level, tile_col, tile_row)).fetchone()[0] + + return mapbox_vector_tile.decode(gzip.decompress(tile_data)) + + @async def reload(self): - self.con = sqlite3.connect(self.source.resolve()) + con = sqlite3.connect(self.source.resolve()) - self.metadata = dict(self.con.execute('SELECT name, value FROM metadata').fetchall()) + self.metadata = dict(con.execute('SELECT name, value FROM metadata').fetchall()) - tiles = self.con.execute('SELECT zoom_level, tile_column, tile_row FROM tiles') - self.rows = tiles.fetchall() + tiles = con.execute('SELECT zoom_level, tile_column, tile_row FROM tiles') + for r in Progress(tiles.fetchall()): + self.addRow(r) class PbfSheet(Sheet): columns = [ ColumnItem('layer', 0), Column('geometry_type', getter=lambda col,row: row[1]['geometry']['type']), - Column('geometry_coords', getter=lambda col,row: row[1]['geometry']['coordinates']), - Column('geometry_coords_depth', getter=lambda col,row: getListDepth(row[1]['geometry']['coordinates'])), + Column('geometry_coords', getter=lambda col,row: row[1]['geometry']['coordinates'], width=0), + Column('geometry_coords_depth', getter=lambda col,row: getListDepth(row[1]['geometry']['coordinates']), width=0), ] + nKeys = 1 # layer commands = [ Command('.', 'vd.push(PbfCanvas(name+"_map", source=sheet, sourceRows=[cursorRow]))', 'plot this row only'), Command('g.', 'vd.push(PbfCanvas(name+"_map", source=sheet, sourceRows=selectedRows or rows))', 'plot as map'), @@ -75,7 +78,7 @@ class PbfSheet(Sheet): def reload(self): props = set() # property names self.rows = [] - for r in getFeatures(self.tile_data): + for r in getFeatures(self.source.getTile(*self.sourceRow)): self.rows.append(r) props.update(r[1]['properties'].keys()) @@ -83,32 +86,38 @@ def reload(self): self.addColumn(Column(key, getter=lambda col,row,key=key: row[1]['properties'][key])) -class PbfCanvas(InvertedYGridCanvas): +class PbfCanvas(InvertedCanvas): aspectRatio = 1.0 + def iterpolylines(self, r): + layername, feat = r + geom = feat['geometry'] + t = geom['type'] + coords = geom['coordinates'] + key = self.source.rowkey(r) + + if t == 'LineString': + yield coords, self.plotColor(key), r + elif t == 'Point': + yield [coords], self.plotColor(key), r + elif t == 'Polygon': + for poly in coords: + yield poly+[poly[0]], self.plotColor(key), r + elif t == 'MultiLineString': + for line in coords: + yield line, self.plotColor(key), r + elif t == 'MultiPolygon': + for mpoly in coords: + for poly in mpoly: + yield poly+[poly[0]], self.plotColor(key), r + else: + assert False, t + @async def reload(self): + self.reset() + for r in Progress(self.sourceRows): - layername, feat = r - geom = feat['geometry'] - t = geom['type'] - coords = geom['coordinates'] - - if t == 'LineString': - self.polyline(coords, self.plotColor((layername,)), r) - elif t == 'Point': - x, y = coords - self.point(x, y, self.plotColor((layername,)), r) - elif t == 'Polygon': - for poly in coords: - self.polygon(poly, self.plotColor((layername,)), r) - elif t == 'MultiLineString': - for line in coords: - self.polyline(line, self.plotColor((layername,)), r) - elif t == 'MultiPolygon': - for mpoly in coords: - for poly in mpoly: - self.polygon(poly, self.plotColor((layername,)), r) - else: - assert False, t + for vertexes, attr, row in self.iterpolylines(r): + self.polyline(vertexes, attr, row) self.refresh() diff --git a/visidata/loaders/postgres.py b/visidata/loaders/postgres.py index 401383ce9..c12eb0841 100644 --- a/visidata/loaders/postgres.py +++ b/visidata/loaders/postgres.py @@ -13,7 +13,7 @@ def codeToType(type_code, colname): return anytype -def openurl_postgres(url): +def openurl_postgres(url, filetype=None): import psycopg2 dbname = url.path[1:] diff --git a/visidata/loaders/shp.py b/visidata/loaders/shp.py index 14b968cd7..c7ce6714d 100644 --- a/visidata/loaders/shp.py +++ b/visidata/loaders/shp.py @@ -45,12 +45,11 @@ def reload(self): self.addRow(shaperec) -class ShapeMap(GridCanvas): +class ShapeMap(Canvas): aspectRatio = 1.0 @async def reload(self): - self.gridlines.clear() self.reset() for row in Progress(self.sourceRows): diff --git a/visidata/man/vd.1 b/visidata/man/vd.1 index b1f6f6372..a469c426b 100644 --- a/visidata/man/vd.1 +++ b/visidata/man/vd.1 @@ -59,7 +59,7 @@ replays in batch mode (with no interface) .Bl -tag -width XXXXXXXXXXXXXXXXXXX -compact -offset XXX .It Sy ^U pauses/resumes replay -.It Sy Space +.It Sy Tab executes next row in replaying sheet .It Sy ^K cancels current replay @@ -77,7 +77,7 @@ views all commands available on current sheet .It Ic ^Q aborts program immediately .It Ic ^C -cancels user input or current task +cancels user input or aborts all async threads on current sheet .It Ic " q" quits current sheet .It Ic "gq" @@ -147,38 +147,37 @@ adjusts width of all visible columns hides current column (to unhide, go to .Sy C Ns olumns sheet and Sy e Ns dit its width) .It Ic "z-" Ns -cuts width of current column in half +reduces width of current column by half .Pp .It Ic \&! Ns pins current column on the left as a key column -.It Ic ^ -edits name of current column .It Ic "~ # % $ @" -sets type of current column to str/int/float/currency/date +sets type of current column to untyped/int/float/currency/date +.It Ic " ^" +edits name of current column +.It Ic " g^" +sets names of all unnamed visible columns to contents of selected rows (or current row) +.It Ic " z^" +sets name of current column to contents of current cell +.It Ic "gz^" +sets name of current column to combined contents of current column for selected rows (or current row) .Pp .It Ic " =" Ar expr .No creates new column from Python Ar expr Ns , with column names as variables .It Ic " g=" Ar expr .No sets current column for selected rows to result of Python Ar expr -.It Ic "gz=" Ar iterable -.No sets selected rows in current column to the results of a Python Ar iterable +.It Ic "gz=" Ar expr +.No sets current column for selected rows to the items in result of Python sequence Ar expr +.It Ic " z=" Ar expr +.No sets current cell to result of evaluated Python Ar expr No on current row .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -.It Ic " z=" Ar expr -.No evaluates Python Ar expr No on current row and displays result on status line -.Pp -.It Ic " g^" -sets names of all visible columns to contents of current row -.It Ic " z^" -sets name of current column to contents of current cell -.It Ic "gz^" -sets name of current column to combined contents of current cell in selected rows -.Pp .It Ic " '" Ns " (tick)" adds a frozen copy of current column with all cells evaluated -. .It Ic "g'" opens a frozen copy of current sheet with all visible columns evaluated +.It Ic "z' gz'" +adds/resets cache for current/all visible column(s) .Pp .It Ic "\&:" Ar regex .No adds new columns from Ar regex No split; number of columns determined by example row at cursor @@ -229,14 +228,18 @@ opens duplicate sheet with deepcopy of selected rows appends a blank row .It Ic " ga" Ar number .No appends Ar number No blank rows -.It Ic " d gd" +.It Ic " d gd" deletes current/all selected row(s) and moves to clipboard -.It Ic " y gy" +.It Ic " y gy" copies current/all selected row(s) to clipboard -.It Ic " p Shift-P" -pastes most recent clipboard rows after/before current row +.It Ic " zy" +copies current cell to clipboard +.It Ic " p P" +pastes clipboard rows after/before current row +.It Ic " zp gzp" +sets contents of current column for current/all selected row(s) to last clipboard value .It Ic " f" -fills null cells in current column with content of non-null cells up the current column +fills null cells in current column with contents of non-null cells up the current column . . .It Ic " e" Ar text @@ -287,10 +290,10 @@ opens .It Ic "^S" Ar filename .No saves current sheet to Ar filename No in format determined by extension (default .tsv) .It Ic "^D" Ar filename.vd -.No saves commandlog to Ar filename.vd No file -.It Ic "Shift-A" Ar number +.No saves CommandLog to Ar filename.vd No file +.It Ic "A" Ar number .No opens new blank sheet with Ar number No columns -.It Ic "Shift-R" Ar number +.It Ic "R" Ar number opens duplicate sheet with a random population subset of .Ar number No rows .Pp @@ -303,26 +306,35 @@ opens duplicate sheet with a random population subset of .Ss Data Visualization .Bl -tag -width XXXXXXXXXXXXX -compact .It Ic " ." No (dot) -.No graphs current numeric column vs key columns. Numeric key column is on the x-axis, while categorical key columns determine color. +.No plots current numeric column vs key columns. Numeric key column is used for the x-axis; categorical key column values determine color. .It Ic "g." -.No opens a graph of all visible numeric columns vs key column +.No plots a graph of all visible numeric columns vs key columns. .Pp .El -.No If rows on the current sheet represent plottable coordinates (as in .shp or vector .mbtiles sources), Sy "." No plots the current row, and Ic "g." No plots all selected rows (or all rows if none selected). +.No If rows on the current sheet represent plottable coordinates (as in .shp or vector .mbtiles sources), +.Ic " ." No plots the current row, and Ic "g." No plots all selected rows (or all rows if none selected). .Ss " Canvas-specific Commands" .Bl -tag -width XXXXXXXXXXXXXXXXXX -compact -offset XXX .It Ic " + -" -increase/decrease zoomlevel, centered on cursor -.It Ic " s u" -selects/unselects rows on source sheet contained within canvas cursor -.It Ic "gs gu" -selects/unselects rows visible on the screen -.It Ic "Enter" -pushes source sheet with only rows contained within canvas cursor -.It Ic "1" No - Ic "9" +increases/decreases zoomlevel, centered on cursor +.It Ic " _" No (underscore) +zooms to fit full extent +.It Ic " s t u" +selects/toggles/unselects rows on source sheet contained within canvas cursor +.It Ic "gs gt gu" +selects/toggles/unselects rows visible on screen +.It Ic " Enter" +opens sheet of source rows contained within canvas cursor +.It Ic "gEnter" +opens sheet of source rows visible on screen +.It Ic " 1" No - Ic "9" toggles display of layers +.It Ic "^L" +redraws all pixels on canvas +.It Ic " w" +.No toggles Ic show_graph_labels No option .It Ic "mouse scrollwheel" -zooms in/out on a canvas +zooms in/out of canvas .It Ic "left click-drag" sets canvas cursor .It Ic "right click-drag" @@ -332,12 +344,12 @@ scrolls canvas . .Bl -tag -width XXXXXXXXXXXXXXX -compact . -.It Ic Shift-V +.It Ic V views contents of current cell in a new TextSheet .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset xxx -.It Ic "w" -toggles text wrap (only on TextSheet) +.It Ic "v" +toggles visibility (text wrap on TextSheet, legends/axes on Graph) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact .It Ic " ^^" No (Ctrl-^) @@ -359,17 +371,21 @@ reloads current sheet .It Ic "z^R" clears cache for current column .It Ic " ^Z" -sends a SIGSTOP +suspends VisiData process +.It Ic " ^P" +.No opens Sy Status History . .El .Pp .Bl -tag -width XXXXXXXXXXXXXXX -compact -.It Ic "^P" -.No opens Sy Status History -.It Ic "^X" -evalutes Python expression and opens sheet for browsing resulting Python object -.It Ic "^Y z^Y" -opens sheet of current row/cell as Python object +.It Ic " ^Y z^Y g^Y" +opens current row/cell/sheet as Python object +.It Ic " ^X" Ar expr +.No evaluates Python Ar expr No and opens result as python object +.It Ic "z^X" Ar expr +.No evaluates Python Ar expr No on current row and shows result on status line +.It Ic "g^X" Ar stmt +.No executes Python Ar stmt No in the global scope .El . .Ss Internal Sheets List @@ -389,7 +405,7 @@ opens sheet of current row/cell as Python object .It Sy " \&." .Sy Status History No (^P) " view history of status messages" .It Sy " \&." -.Sy Tasks Sheet No (^T) " view, cancel, and profile asynchronous tasks" +.Sy Threads Sheet No (^T) " view, cancel, and profile asynchronous threads" .Pp .It Sy Derived Sheets .It Sy " \&." @@ -405,15 +421,20 @@ opens sheet of current row/cell as Python object .Ss Columns Sheet (Shift-C) .Bl -inset -compact .It Properties of columns on the source sheet can be changed with standard editing commands ( Ns Sy e ge g= Del Ns ) on the Sy Columns Sheet Ns . Multiple aggregators can be set by listing them (separated by spaces) in the aggregators column. The 'g' commands affect the selected rows, which are actually the literal columns on the source sheet. +.It (global commands) +.El +.Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX +.It Ic gC +.No opens Sy Columns Sheet No with all columns from all sheets .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic " &" adds column from concatenating selected source columns .It Ic " !" -toggles current column as key on source sheet +pins current column on the left as a key column on source sheet .It Ic "g!" -toggles selected columns as keys on source sheet +pins selected columns on the left as key columns on source sheet .It Ic "g+" adds aggregator to selected source columns .It Ic "g_" No (underscore) @@ -421,9 +442,11 @@ adjusts widths of selected columns on source sheet .It Ic "g-" No (hyphen) hides selected columns on source sheet .It Ic " ~ # % $ @" -sets type of current column to str/int/float/currency/date +sets type of current column on source sheet to str/int/float/currency/date .It Ic "g~ g# g% g$ g@" -sets type of selected columns to str/int/float/currency/date +sets type of selected columns on source sheet to str/int/float/currency/date +.It Ic "z~ gz~" +sets type of current/selected column(s) on source sheet to anytype .It Ic " Enter" .No opens a Sy Frequency Table No sheet grouped on column referenced in current row .El @@ -435,6 +458,8 @@ sets type of selected columns to str/int/float/currency/date .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic "Enter" jumps to sheet referenced in current row +.It Ic "gC" +.No opens Sy Columns Sheet No with all columns from selected sheets .It Ic "&" Ar jointype .No merges selected sheets with visible columns from all, keeping rows according to Ar jointype Ns : .El @@ -460,17 +485,17 @@ jumps to sheet referenced in current row edits option .El . -.Ss Commandlog (Shift-D) +.Ss CommandLog (Shift-D) .Bl -inset -compact .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX -.It Ic " ^Z" -sends a SIGSTOP .It Ic " x" replays command in current row .It Ic "gx" -replays contents of entire commandlog +replays contents of entire CommandLog +.It Ic " ^C" +aborts replay .El . .Ss DERIVED SHEETS @@ -481,12 +506,14 @@ replays contents of entire commandlog .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic gF -opens a Frequency Table, grouped by all key columns on source sheet +opens Frequency Table, grouped by all key columns on source sheet .It Ic zF -opens a one-line summary for selected rows +opens one-line summary for selected rows .It (sheet-specific commands) .It Ic " s t u" selects/toggles/unselects these entries in source sheet +.It Ic " Enter" +opens sheet of source rows which are grouped in current cell .El . .Ss Describe Sheet (Shift-I) @@ -494,8 +521,10 @@ selects/toggles/unselects these entries in source sheet .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX -.It Ic "zs zt zu" -selects/toggles/unselects rows on source sheet which are being described in current cell +.It Ic "zs zu" +selects/unselects rows on source sheet which are being described in current cell +.It Ic " !" +pins current column on the left as a key column on source sheet .It Ic " Enter" .No opens a Sy Frequency Table No sheet grouped on column referenced in current row .It Ic "zEnter" @@ -509,15 +538,15 @@ opens copy of source sheet with rows described in current cell .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic " Enter" -opens sheet of source rows which comprise current pivot row +opens sheet of source rows aggregated in current pivot row .It Ic "zEnter" -opens sheet of source rows which comprise current pivot cell +opens sheet of source rows aggregated in current pivot cell .El .Ss Melted Sheet (Shift-M) .Bl -inset -compact .It Opens melted sheet (unpivot), with all non-key columns reduced to Variable-Value rows. .El -.Ss Python Object Sheet (^X ^Y) +.Ss Python Object Sheet (^X ^Y g^Y z^Y) .Bl -inset -compact .It (sheet-specific commands) .El @@ -579,6 +608,8 @@ exit on error and display stacktrace curses timeout in ms .It Sy --force-256-colors Ns = Ns Ar "bool " No "False" use 256 colors even if curses reports fewer +.It Sy --use-default-colors Ns = Ns Ar "bool " No "False" +set curses to use default terminal colors .It Sy --note-pending Ns = Ns Ar "str " No "\[u231B]" note to display for pending cells .It Sy --note-format-exc Ns = Ns Ar "str " No "?" @@ -589,8 +620,8 @@ cell note for an exception during computation amount to scroll with scrollwheel .It Sy --skip Ns = Ns Ar "int " No "0" skip first N lines of text input -.It Sy --profile-tasks Ns = Ns Ar "bool " No "True" -profile async tasks +.It Sy --profile-threads Ns = Ns Ar "bool " No "True" +profile async threads .It Sy --min-memory-mb Ns = Ns Ar "int " No "0" minimum memory to continue loading and async processing .It Sy --confirm-overwrite Ns = Ns Ar "bool " No "True" @@ -607,6 +638,8 @@ show methods and _private properties time to wait between replayed commands, in seconds .It Sy --replay-movement Ns = Ns Ar "bool " No "False" insert movements during replay +.It Sy --rowkey-prefix Ns = Ns Ar "str " No "\[u30AD]" +string prefix for rowkey in the cmdlog .It Sy --regex-maxsplit Ns = Ns Ar "int " No "0" maxsplit to pass to regex.split .It Sy --show-graph-labels Ns = Ns Ar "bool " No "True" @@ -700,6 +733,8 @@ status indicator for active replay status indicator for paused replay .It Sy "disp_pixel_random " No "False" randomly choose attr from set of pixels instead of most common +.It Sy "color_graph_hidden " No "238 blue" +color of legend for hidden attribute .It Sy "color_graph_axis " No "bold" color for graph axis labels .El @@ -735,7 +770,7 @@ For example: . .Sh SUPPORTED SOURCES These are the supported sources: - +. .Bl -inset -compact -offset xxx .It Sy tsv No (tab-separated value) .Bl -inset -compact -offset xxx @@ -759,6 +794,13 @@ These are the supported sources: .El .El .Pp +.Bl -inset -compact -offset xxx +.It Sy json No (javascript object notation) +.Bl -inset -compact -offset xxx +.It Nested components of .json files are represented as cells containing lists ( Ns Sy [] Ns ). They can be further expanded with Sy z^Y. +.El +.El +.Pp .Bl -inset -compact .It For these multi-table sources, the first sheet is a directory of tables. .It Sy Enter No loads the entire table into memory. @@ -772,6 +814,8 @@ These are the supported sources: .It Sy postgres No (requires Sy psycopg2 Ns ) .It Sy shp No (requires Sy pyshp Ns ) .It Sy mbtiles No (vector only, requires Sy mapbox-vector-tile Ns ) +.It Sy http No (requires Sy requests Ns ) +.It Sy html No (requires Sy lxml Ns ) .El . .Sh AUTHOR diff --git a/visidata/man/vd.inc b/visidata/man/vd.inc index 98effd1ae..aec16acce 100644 --- a/visidata/man/vd.inc +++ b/visidata/man/vd.inc @@ -59,7 +59,7 @@ replays in batch mode (with no interface) .Bl -tag -width XXXXXXXXXXXXXXXXXXX -compact -offset XXX .It Sy ^U pauses/resumes replay -.It Sy Space +.It Sy Tab executes next row in replaying sheet .It Sy ^K cancels current replay @@ -77,7 +77,7 @@ views all commands available on current sheet .It Ic ^Q aborts program immediately .It Ic ^C -cancels user input or current task +cancels user input or aborts all async threads on current sheet .It Ic " q" quits current sheet .It Ic "gq" @@ -147,38 +147,37 @@ adjusts width of all visible columns hides current column (to unhide, go to .Sy C Ns olumns sheet and Sy e Ns dit its width) .It Ic "z-" Ns -cuts width of current column in half +reduces width of current column by half .Pp .It Ic \&! Ns pins current column on the left as a key column -.It Ic ^ -edits name of current column .It Ic "~ # % $ @" -sets type of current column to str/int/float/currency/date +sets type of current column to untyped/int/float/currency/date +.It Ic " ^" +edits name of current column +.It Ic " g^" +sets names of all unnamed visible columns to contents of selected rows (or current row) +.It Ic " z^" +sets name of current column to contents of current cell +.It Ic "gz^" +sets name of current column to combined contents of current column for selected rows (or current row) .Pp .It Ic " =" Ar expr .No creates new column from Python Ar expr Ns , with column names as variables .It Ic " g=" Ar expr .No sets current column for selected rows to result of Python Ar expr -.It Ic "gz=" Ar iterable -.No sets selected rows in current column to the results of a Python Ar iterable +.It Ic "gz=" Ar expr +.No sets current column for selected rows to the items in result of Python sequence Ar expr +.It Ic " z=" Ar expr +.No sets current cell to result of evaluated Python Ar expr No on current row .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -.It Ic " z=" Ar expr -.No evaluates Python Ar expr No on current row and displays result on status line -.Pp -.It Ic " g^" -sets names of all visible columns to contents of current row -.It Ic " z^" -sets name of current column to contents of current cell -.It Ic "gz^" -sets name of current column to combined contents of current cell in selected rows -.Pp .It Ic " '" Ns " (tick)" adds a frozen copy of current column with all cells evaluated -. .It Ic "g'" opens a frozen copy of current sheet with all visible columns evaluated +.It Ic "z' gz'" +adds/resets cache for current/all visible column(s) .Pp .It Ic "\&:" Ar regex .No adds new columns from Ar regex No split; number of columns determined by example row at cursor @@ -229,14 +228,18 @@ opens duplicate sheet with deepcopy of selected rows appends a blank row .It Ic " ga" Ar number .No appends Ar number No blank rows -.It Ic " d gd" +.It Ic " d gd" deletes current/all selected row(s) and moves to clipboard -.It Ic " y gy" +.It Ic " y gy" copies current/all selected row(s) to clipboard -.It Ic " p Shift-P" -pastes most recent clipboard rows after/before current row +.It Ic " zy" +copies current cell to clipboard +.It Ic " p P" +pastes clipboard rows after/before current row +.It Ic " zp gzp" +sets contents of current column for current/all selected row(s) to last clipboard value .It Ic " f" -fills null cells in current column with content of non-null cells up the current column +fills null cells in current column with contents of non-null cells up the current column . . .It Ic " e" Ar text @@ -287,10 +290,10 @@ opens .It Ic "^S" Ar filename .No saves current sheet to Ar filename No in format determined by extension (default .tsv) .It Ic "^D" Ar filename.vd -.No saves commandlog to Ar filename.vd No file -.It Ic "Shift-A" Ar number +.No saves CommandLog to Ar filename.vd No file +.It Ic "A" Ar number .No opens new blank sheet with Ar number No columns -.It Ic "Shift-R" Ar number +.It Ic "R" Ar number opens duplicate sheet with a random population subset of .Ar number No rows .Pp @@ -303,26 +306,35 @@ opens duplicate sheet with a random population subset of .Ss Data Visualization .Bl -tag -width XXXXXXXXXXXXX -compact .It Ic " ." No (dot) -.No graphs current numeric column vs key columns. Numeric key column is on the x-axis, while categorical key columns determine color. +.No plots current numeric column vs key columns. Numeric key column is used for the x-axis; categorical key column values determine color. .It Ic "g." -.No opens a graph of all visible numeric columns vs key column +.No plots a graph of all visible numeric columns vs key columns. .Pp .El -.No If rows on the current sheet represent plottable coordinates (as in .shp or vector .mbtiles sources), Sy "." No plots the current row, and Ic "g." No plots all selected rows (or all rows if none selected). +.No If rows on the current sheet represent plottable coordinates (as in .shp or vector .mbtiles sources), +.Ic " ." No plots the current row, and Ic "g." No plots all selected rows (or all rows if none selected). .Ss " Canvas-specific Commands" .Bl -tag -width XXXXXXXXXXXXXXXXXX -compact -offset XXX .It Ic " + -" -increase/decrease zoomlevel, centered on cursor -.It Ic " s u" -selects/unselects rows on source sheet contained within canvas cursor -.It Ic "gs gu" -selects/unselects rows visible on the screen -.It Ic "Enter" -pushes source sheet with only rows contained within canvas cursor -.It Ic "1" No - Ic "9" +increases/decreases zoomlevel, centered on cursor +.It Ic " _" No (underscore) +zooms to fit full extent +.It Ic " s t u" +selects/toggles/unselects rows on source sheet contained within canvas cursor +.It Ic "gs gt gu" +selects/toggles/unselects rows visible on screen +.It Ic " Enter" +opens sheet of source rows contained within canvas cursor +.It Ic "gEnter" +opens sheet of source rows visible on screen +.It Ic " 1" No - Ic "9" toggles display of layers +.It Ic "^L" +redraws all pixels on canvas +.It Ic " w" +.No toggles Ic show_graph_labels No option .It Ic "mouse scrollwheel" -zooms in/out on a canvas +zooms in/out of canvas .It Ic "left click-drag" sets canvas cursor .It Ic "right click-drag" @@ -332,12 +344,12 @@ scrolls canvas . .Bl -tag -width XXXXXXXXXXXXXXX -compact . -.It Ic Shift-V +.It Ic V views contents of current cell in a new TextSheet .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset xxx -.It Ic "w" -toggles text wrap (only on TextSheet) +.It Ic "v" +toggles visibility (text wrap on TextSheet, legends/axes on Graph) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact .It Ic " ^^" No (Ctrl-^) @@ -359,17 +371,21 @@ reloads current sheet .It Ic "z^R" clears cache for current column .It Ic " ^Z" -sends a SIGSTOP +suspends VisiData process +.It Ic " ^P" +.No opens Sy Status History . .El .Pp .Bl -tag -width XXXXXXXXXXXXXXX -compact -.It Ic "^P" -.No opens Sy Status History -.It Ic "^X" -evalutes Python expression and opens sheet for browsing resulting Python object -.It Ic "^Y z^Y" -opens sheet of current row/cell as Python object +.It Ic " ^Y z^Y g^Y" +opens current row/cell/sheet as Python object +.It Ic " ^X" Ar expr +.No evaluates Python Ar expr No and opens result as python object +.It Ic "z^X" Ar expr +.No evaluates Python Ar expr No on current row and shows result on status line +.It Ic "g^X" Ar stmt +.No executes Python Ar stmt No in the global scope .El . .Ss Internal Sheets List @@ -389,7 +405,7 @@ opens sheet of current row/cell as Python object .It Sy " \&." .Sy Status History No (^P) " view history of status messages" .It Sy " \&." -.Sy Tasks Sheet No (^T) " view, cancel, and profile asynchronous tasks" +.Sy Threads Sheet No (^T) " view, cancel, and profile asynchronous threads" .Pp .It Sy Derived Sheets .It Sy " \&." @@ -405,15 +421,20 @@ opens sheet of current row/cell as Python object .Ss Columns Sheet (Shift-C) .Bl -inset -compact .It Properties of columns on the source sheet can be changed with standard editing commands ( Ns Sy e ge g= Del Ns ) on the Sy Columns Sheet Ns . Multiple aggregators can be set by listing them (separated by spaces) in the aggregators column. The 'g' commands affect the selected rows, which are actually the literal columns on the source sheet. +.It (global commands) +.El +.Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX +.It Ic gC +.No opens Sy Columns Sheet No with all columns from all sheets .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic " &" adds column from concatenating selected source columns .It Ic " !" -toggles current column as key on source sheet +pins current column on the left as a key column on source sheet .It Ic "g!" -toggles selected columns as keys on source sheet +pins selected columns on the left as key columns on source sheet .It Ic "g+" adds aggregator to selected source columns .It Ic "g_" No (underscore) @@ -421,9 +442,11 @@ adjusts widths of selected columns on source sheet .It Ic "g-" No (hyphen) hides selected columns on source sheet .It Ic " ~ # % $ @" -sets type of current column to str/int/float/currency/date +sets type of current column on source sheet to str/int/float/currency/date .It Ic "g~ g# g% g$ g@" -sets type of selected columns to str/int/float/currency/date +sets type of selected columns on source sheet to str/int/float/currency/date +.It Ic "z~ gz~" +sets type of current/selected column(s) on source sheet to anytype .It Ic " Enter" .No opens a Sy Frequency Table No sheet grouped on column referenced in current row .El @@ -435,6 +458,8 @@ sets type of selected columns to str/int/float/currency/date .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic "Enter" jumps to sheet referenced in current row +.It Ic "gC" +.No opens Sy Columns Sheet No with all columns from selected sheets .It Ic "&" Ar jointype .No merges selected sheets with visible columns from all, keeping rows according to Ar jointype Ns : .El @@ -460,17 +485,17 @@ jumps to sheet referenced in current row edits option .El . -.Ss Commandlog (Shift-D) +.Ss CommandLog (Shift-D) .Bl -inset -compact .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX -.It Ic " ^Z" -sends a SIGSTOP .It Ic " x" replays command in current row .It Ic "gx" -replays contents of entire commandlog +replays contents of entire CommandLog +.It Ic " ^C" +aborts replay .El . .Ss DERIVED SHEETS @@ -481,12 +506,14 @@ replays contents of entire commandlog .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic gF -opens a Frequency Table, grouped by all key columns on source sheet +opens Frequency Table, grouped by all key columns on source sheet .It Ic zF -opens a one-line summary for selected rows +opens one-line summary for selected rows .It (sheet-specific commands) .It Ic " s t u" selects/toggles/unselects these entries in source sheet +.It Ic " Enter" +opens sheet of source rows which are grouped in current cell .El . .Ss Describe Sheet (Shift-I) @@ -494,8 +521,10 @@ selects/toggles/unselects these entries in source sheet .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX -.It Ic "zs zt zu" -selects/toggles/unselects rows on source sheet which are being described in current cell +.It Ic "zs zu" +selects/unselects rows on source sheet which are being described in current cell +.It Ic " !" +pins current column on the left as a key column on source sheet .It Ic " Enter" .No opens a Sy Frequency Table No sheet grouped on column referenced in current row .It Ic "zEnter" @@ -509,15 +538,15 @@ opens copy of source sheet with rows described in current cell .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic " Enter" -opens sheet of source rows which comprise current pivot row +opens sheet of source rows aggregated in current pivot row .It Ic "zEnter" -opens sheet of source rows which comprise current pivot cell +opens sheet of source rows aggregated in current pivot cell .El .Ss Melted Sheet (Shift-M) .Bl -inset -compact .It Opens melted sheet (unpivot), with all non-key columns reduced to Variable-Value rows. .El -.Ss Python Object Sheet (^X ^Y) +.Ss Python Object Sheet (^X ^Y g^Y z^Y) .Bl -inset -compact .It (sheet-specific commands) .El @@ -587,7 +616,7 @@ For example: . .Sh SUPPORTED SOURCES These are the supported sources: - +. .Bl -inset -compact -offset xxx .It Sy tsv No (tab-separated value) .Bl -inset -compact -offset xxx @@ -611,6 +640,13 @@ These are the supported sources: .El .El .Pp +.Bl -inset -compact -offset xxx +.It Sy json No (javascript object notation) +.Bl -inset -compact -offset xxx +.It Nested components of .json files are represented as cells containing lists ( Ns Sy [] Ns ). They can be further expanded with Sy z^Y. +.El +.El +.Pp .Bl -inset -compact .It For these multi-table sources, the first sheet is a directory of tables. .It Sy Enter No loads the entire table into memory. @@ -624,6 +660,8 @@ These are the supported sources: .It Sy postgres No (requires Sy psycopg2 Ns ) .It Sy shp No (requires Sy pyshp Ns ) .It Sy mbtiles No (vector only, requires Sy mapbox-vector-tile Ns ) +.It Sy http No (requires Sy requests Ns ) +.It Sy html No (requires Sy lxml Ns ) .El . .Sh AUTHOR diff --git a/visidata/metasheets.py b/visidata/metasheets.py index 8c5da017c..8df6a90ce 100644 --- a/visidata/metasheets.py +++ b/visidata/metasheets.py @@ -19,11 +19,12 @@ def createJoinedSheet(sheets, jointype=''): SheetsSheet.commands += [ Command('&', 'vd.replace(createJoinedSheet(selectedRows, jointype=chooseOne(jointypes)))', 'merge the selected sheets with visible columns from all, keeping rows according to jointype'), + Command('gC', 'vd.push(ColumnsSheet("all_columns", source=selectedRows or rows[1:]))', 'open Columns Sheet with all columns from selected sheets'), ] SheetsSheet.columns.insert(1, ColumnAttr('progressPct')) -# used on both ColumnsSheet and DescribeSheet, affecting the 'row' (source column) +# used ColumnsSheet, affecting the 'row' (source column) columnCommands = [ Command('_', 'cursorRow.width = cursorRow.getMaxWidth(source.visibleRows)', 'adjust width of source column'), Command('-', 'cursorRow.width = 0', 'hide source column on source sheet'), @@ -31,9 +32,10 @@ def createJoinedSheet(sheets, jointype=''): Command('#', 'cursorRow.type = int', 'set type of source column to int'), Command('@', 'cursorRow.type = date', 'set type of source column to date'), Command('$', 'cursorRow.type = currency', 'set type of source column to currency'), - Command('~', 'cursorRow.type = str', 'set type of source column to str'), + Command('~', 'cursorRow.type = str', 'set type of current column to str'), + Command('z~', 'cursorRow.type = anytype', 'set type of current column to anytype'), - Command('g!', 'for c in selectedRows or [cursorRow]: source.toggleKeyColumn(source.columns.index(c))', 'toggle selected columns as keys on source sheet'), + Command('g!', 'for c in selectedRows or [cursorRow]: source.toggleKeyColumn(source.columns.index(c))', 'pin selected columns on the left as key columns on source sheet'), Command('g-', 'for c in selectedRows or source.nonKeyVisibleCols: c.width = 0', 'hide selected source columns on source sheet'), Command('g_', 'for c in selectedRows or source.nonKeyVisibleCols: c.width = c.getMaxWidth(source.visibleRows)', 'adjust widths of selected source columns'), Command('g%', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = float', 'set type of selected source columns to float'), @@ -41,10 +43,11 @@ def createJoinedSheet(sheets, jointype=''): Command('g@', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = date', 'set type of selected source columns to date'), Command('g$', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = currency', 'set type of selected columns to currency'), Command('g~', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = str', 'set type of selected columns to str'), + Command('gz~', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = anytype', 'set type of selected columns to anytype'), ] ColumnsSheet.commands += columnCommands + [ - Command('!', 'source.toggleKeyColumn(cursorRowIndex)', 'toggle column as key on source sheet'), + Command('!', 'source.toggleKeyColumn(cursorRowIndex)', 'pin current column on the left as a key column on source sheet'), Command('&', 'rows.insert(cursorRowIndex, combineColumns(selectedRows))', 'add column from concatenating selected source columns'), ] DescribeSheet.commands += columnCommands diff --git a/visidata/pivot.py b/visidata/pivot.py index 47122be3b..9a166d8da 100644 --- a/visidata/pivot.py +++ b/visidata/pivot.py @@ -8,9 +8,9 @@ class SheetPivot(Sheet): rowtype = 'aggregated rows' commands = [ Command('z'+ENTER, 'vs=copy(source); vs.name+="_%s"%cursorCol.aggvalue; vs.rows=cursorRow[1].get(cursorCol.aggvalue, []); vd.push(vs)', - 'push sheet of source rows aggregated in this cell'), + 'open sheet of source rows aggregated in current cell'), Command(ENTER, 'vs=copy(source); vs.name+="_%s"%"+".join(cursorRow[0]); vs.rows=sum(cursorRow[1].values(), []); vd.push(vs)', - 'push sheet of source rows aggregated in this cell') + 'open sheet of source rows aggregated in current cell') ] def __init__(self, srcsheet, variableCols): super().__init__(srcsheet.name+'_pivot_'+''.join(c.name for c in variableCols), source=srcsheet) diff --git a/visidata/pyobj.py b/visidata/pyobj.py index 5320f7133..7441518f4 100644 --- a/visidata/pyobj.py +++ b/visidata/pyobj.py @@ -2,11 +2,13 @@ option('pyobj_show_hidden', False, 'show methods and _private properties') -globalCommand('^X', 'expr = input("eval: ", "expr"); push_pyobj(expr, eval(expr))', 'evaluate Python expression and open sheet for browsing resulting Python object') -globalCommand('g^X', 'expr = input("exec: ", "expr"); exec(expr, getGlobals())', 'execute Python statement in the global scope') +globalCommand('^X', 'expr = input("eval: ", "expr", completer=CompleteExpr()); push_pyobj(expr, eval(expr))', 'evaluate Python expression and open result as Python object') +globalCommand('g^X', 'expr = input("exec: ", "expr", completer=CompleteExpr()); exec(expr, getGlobals())', 'execute Python statement in the global scope') +globalCommand('z^X', 'status(evalexpr(inputExpr("status="), cursorRow))', 'evaluate Python expression on current row and show result on status line') -globalCommand('^Y', 'status(type(cursorRow)); push_pyobj("%s.row[%s]" % (sheet.name, cursorRowIndex), cursorRow)', 'open sheet of current row as Python object') -globalCommand('z^Y', 'status(type(cursorValue)); push_pyobj("%s.row[%s].%s" % (sheet.name, cursorRowIndex, cursorCol.name), cursorValue)', 'open sheet of current cell as Python object') +globalCommand('^Y', 'status(type(cursorRow)); push_pyobj("%s[%s]" % (sheet.name, cursorRowIndex), cursorRow)', 'open current row as Python object') +globalCommand('z^Y', 'status(type(cursorValue)); push_pyobj("%s[%s].%s" % (sheet.name, cursorRowIndex, cursorCol.name), cursorValue)', 'open current cell as Python object') +globalCommand('g^Y', 'status(type(sheet)); push_pyobj(sheet.name+"_sheet", sheet)', 'open current sheet as Python object') # used as ENTER in several pyobj sheets globalCommand('pyobj-dive', 'push_pyobj("%s[%s]" % (name, cursorRowIndex), cursorRow).cursorRowIndex = cursorColIndex', 'dive further into Python object') @@ -58,7 +60,8 @@ def SheetList(name, src, **kwargs): 'Creates a Sheet from a list of homogenous dicts or namedtuples.' if not src: - error('no content') + status('no content in ' + name) + return if isinstance(src[0], dict): return ListOfDictSheet(name, source=src, **kwargs) @@ -146,8 +149,8 @@ class SheetObject(Sheet): rowtype = 'attributes' commands = [ Command(ENTER, 'v = getattr(source, cursorRow); push_pyobj(joinSheetnames(name, cursorRow), v() if callable(v) else v)', 'dive further into Python object'), - Command('e', 'setattr(source, cursorRow, editCell(1)); sheet.cursorRowIndex += 1; reload()', 'edit contents of current cell'), - Command('w', 'options.pyobj_show_hidden = not options.pyobj_show_hidden; reload()', 'toggle whether methods and hidden properties are shown') + Command('e', 'setattr(source, cursorRow, type(getattr(source, cursorRow))(editCell(1))); sheet.cursorRowIndex += 1; reload()', 'edit contents of current cell'), + Command('v', 'options.pyobj_show_hidden = not options.pyobj_show_hidden; reload()', 'toggle whether methods and hidden properties are shown') ] def __init__(self, name, obj, **kwargs): super().__init__(name, source=obj, **kwargs) @@ -171,16 +174,3 @@ def reload(self): self.recalc() self.nKeys = 1 - - -def open_json(p): - 'Handle JSON file as a single object, via `json.load`.' - import json - return load_pyobj(p.name, json.load(p.open_text())) - -# one json object per line -def open_jsonl(p): - 'Handle JSON file as a list of objects, one per line, via `json.loads`.' - import json - return load_pyobj(p.name, list(json.loads(L) for L in p.read_text().splitlines())) - diff --git a/visidata/vdtui.py b/visidata/vdtui.py index c3dd80c5d..3f26582bf 100755 --- a/visidata/vdtui.py +++ b/visidata/vdtui.py @@ -26,7 +26,7 @@ # Just include this whole file in your project as-is. If you do make # modifications, please keep the base vdtui version and append your own id and # version. -__version__ = 'saul.pw/vdtui v0.98.1' +__version__ = 'saul.pw/vdtui v0.99' __author__ = 'Saul Pwanson ' __license__ = 'MIT' __status__ = 'Beta' @@ -53,6 +53,10 @@ class EscapeException(BaseException): 'Inherits from BaseException to avoid "except Exception" clauses. Do not use a blanket "except:" or the task will be uncancelable.' pass +class ExpectedException(Exception): + 'an expected exception' + pass + baseCommands = collections.OrderedDict() # [cmd.name] -> Command baseOptions = collections.OrderedDict() # [opt.name] -> opt @@ -69,10 +73,11 @@ def globalCommand(keystrokes, execstr, helpstr='', longname=None): if longname: cmd = Command(longname, execstr, helpstr) baseCommands[longname] = cmd + assert helpstr or (execstr in baseCommands), 'unknown longname ' + execstr + helpstr = '' for ks in keystrokes: - baseCommands[ks] = Command(ks, longname or execstr, helpstr or 'alias') - assert helpstr or (execstr in baseCommands), 'unknown longname ' + execstr + baseCommands[ks] = Command(ks, longname or execstr, helpstr) def option(name, default, helpstr=''): baseOptions[name] = [name, default, default, helpstr] @@ -128,6 +133,7 @@ def __setitem__(self, k, v): # options[k] = v option('debug', False, 'exit on error and display stacktrace') option('curses_timeout', 100, 'curses timeout in ms') theme('force_256_colors', False, 'use 256 colors even if curses reports fewer') +theme('use_default_colors', False, 'set curses to use default terminal colors') disp_column_fill = ' ' # pad chars after column value theme('disp_none', '', 'visible contents of a cell whose value was None') @@ -191,6 +197,10 @@ def __setitem__(self, k, v): # options[k] = v globalCommand('j', 'move-down') globalCommand('k', 'move-up') globalCommand('l', 'move-right') +globalCommand('gKEY_LEFT', 'move-far-left') +globalCommand('gKEY_RIGHT', 'move-far-right') +globalCommand('gKEY_UP', 'move-top') +globalCommand('gKEY_DOWN', 'move-bottom') globalCommand(['^F', 'kDOWN'], 'move-page-down') globalCommand(['^B', 'kUP'], 'move-page-up') globalCommand(['gg', 'gk'], 'move-top') @@ -203,7 +213,7 @@ def __setitem__(self, k, v): # options[k] = v globalCommand('^L', 'vd.scr.clear()', 'refresh screen') globalCommand('^G', 'status(statusLine)', 'show cursor position and bounds of current sheet on status line') globalCommand('^V', 'status(__version__)', 'show version information on status line') -globalCommand('^P', 'vd.push(TextSheet("statusHistory", vd.statusHistory))', 'open Status History sheet') +globalCommand('^P', 'vd.push(TextSheet("statusHistory", vd.statusHistory, rowtype="statuses"))', 'open Status History') globalCommand('<', 'moveToNextRow(lambda row,sheet=sheet,col=cursorCol,val=cursorValue: col.getValue(row) != val, reverse=True) or status("no different value up this column")', 'move up the current column to the next value') globalCommand('>', 'moveToNextRow(lambda row,sheet=sheet,col=cursorCol,val=cursorValue: col.getValue(row) != val) or status("no different value down this column")', 'move down the current column to the next value') @@ -211,11 +221,12 @@ def __setitem__(self, k, v): # options[k] = v globalCommand('}', 'moveToNextRow(lambda row,sheet=sheet: sheet.isSelected(row)) or status("no next selected row")', 'move down the current column to the next selected row') globalCommand('_', 'cursorCol.toggleWidth(cursorCol.getMaxWidth(visibleRows))', 'adjust width of current column', 'width-curcol-max') -globalCommand('z_', 'cursorCol.width = int(input("set width= ", value=cursorCol.width))', 'set current column width to given value', 'width-curcol-input') +globalCommand('z_', 'cursorCol.width = int(input("set width= ", value=cursorCol.width))', 'adjust current column width to given number', 'width-curcol-input') globalCommand('-', 'cursorCol.width = 0', 'hide current column', 'width-curcol-zero') -globalCommand('z-', 'cursorCol.width = cursorCol.width//2', 'reduce column width by half', 'width-curcol-half') +globalCommand('z-', 'cursorCol.width = cursorCol.width//2', 'reduce width of current column by half', 'width-curcol-half') globalCommand('!', 'toggleKeyColumn(cursorColIndex); cursorRight(+1)', 'pin current column on the left as a key column', 'toggle-curcol-key') +globalCommand('z~', 'cursorCol.type = anytype', 'set type of current column to anytype', 'type-curcol-any') globalCommand('~', 'cursorCol.type = str', 'set type of current column to str', 'type-curcol-str') globalCommand('@', 'cursorCol.type = date', 'set type of current column to date', 'type-curcol-date') globalCommand('#', 'cursorCol.type = int', 'set type of current column to int', 'type-curcol-int') @@ -273,17 +284,18 @@ def __setitem__(self, k, v): # options[k] = v globalCommand(',', 'select(gatherBy(lambda r,c=cursorCol,v=cursorValue: c.getValue(r) == v), progress=False)', 'select rows matching current cell in current column') globalCommand('g,', 'select(gatherBy(lambda r,v=cursorRow: r == v), progress=False)', 'select rows matching current cell in all visible columns') -globalCommand('"', 'vs = copy(sheet); vs.name += "_selectedref"; vs.rows = list(selectedRows or rows); vs.select(vs.rows); vd.push(vs)', 'open duplicate sheet with only selected rows') +globalCommand('"', 'vs = copy(sheet); vs.name += "_selectedref"; vs.rows = list(selectedRows or rows); vs.select(selectedRows); vd.push(vs)', 'open duplicate sheet with only selected rows') globalCommand('g"', 'vs = copy(sheet); vs.name += "_copy"; vs.rows = list(rows); vs.select(selectedRows); vd.push(vs)', 'open duplicate sheet with all rows') -globalCommand('gz"', 'vs = deepcopy(sheet); vs.name += "_selectedcopy"; vs.rows = async_deepcopy(vs, selectedRows or rows); vd.push(vs); status("pushed sheet with async deepcopy of all rows")', 'open duplicate sheet with all rows') +globalCommand('gz"', 'vs = deepcopy(sheet); vs.name += "_selectedcopy"; vs.rows = async_deepcopy(vs, selectedRows or rows); vd.push(vs); status("pushed sheet with async deepcopy of all rows")', 'open duplicate sheet with deepcopy of selected rows') -globalCommand('=', 'addColumn(ColumnExpr(input("new column expr=", "expr")), index=cursorColIndex+1)', 'create new column from Python expression, with column names as variables') -globalCommand('g=', 'cursorCol.setValuesFromExpr(selectedRows or rows, input("set selected=", "expr"))', 'set current column for selected rows to result of Python expression') +globalCommand('=', 'addColumn(ColumnExpr(inputExpr("new column expr=")), index=cursorColIndex+1)', 'create new column from Python expression, with column names as variables') +globalCommand('g=', 'cursorCol.setValuesFromExpr(selectedRows or rows, inputExpr("set selected="))', 'set current column for selected rows to result of Python expression') globalCommand('V', 'vd.push(TextSheet("%s[%s].%s" % (name, cursorRowIndex, cursorCol.name), cursorDisplay.splitlines()))', 'view contents of current cell in a new sheet') globalCommand('S', 'vd.push(SheetsSheet("sheets"))', 'open Sheets Sheet') globalCommand('C', 'vd.push(ColumnsSheet(sheet.name+"_columns", source=sheet))', 'open Columns Sheet') +globalCommand('gC', 'vd.push(ColumnsSheet("all_columns", source=vd.sheets))', 'open Columns Sheet with all columns from all sheets') globalCommand('O', 'vd.push(vd.optionsSheet)', 'open Options') globalCommand(['KEY_F(1)', 'z?'], 'vd.push(HelpSheet(name + "_commands", source=sheet))', 'view VisiData man page', 'help-commands') globalCommand('^Z', 'suspend()', 'suspend VisiData process') @@ -369,8 +381,9 @@ def joinSheetnames(*sheetnames): return '_'.join(str(x) for x in sheetnames) def error(s): - 'Return custom exception as function, for use with `lambda` and `eval`.' - raise Exception(s) + 'Raise an expection.' + status(s) + raise ExpectedException(s) def status(*args): 'Return status property via function call.' @@ -404,8 +417,8 @@ def vd(): 'Return VisiData singleton, which contains all global context' return VisiData() -def exceptionCaught(status=True): - return vd().exceptionCaught(status) +def exceptionCaught(e, **kwargs): + return vd().exceptionCaught(e, **kwargs) def stacktrace(): return traceback.format_exc().strip().splitlines() @@ -510,8 +523,8 @@ def callHook(self, hookname, *args, **kwargs): for f in self.hooks[hookname]: try: r.append(f(*args, **kwargs)) - except Exception: - exceptionCaught() + except Exception as e: + exceptionCaught(e) return r def addThread(self, t, endTime=None): @@ -543,8 +556,7 @@ def toplevelTryFunc(self, func, *args, **kwargs): t.status += 'aborted by user' status('%s aborted' % t.name) except Exception as e: - t.status += status('%s: %s' % (type(e).__name__, ' '.join(str(x) for x in e.args))) - exceptionCaught() + exceptionCaught(e) t.sheet.currentThreads.remove(t) return ret @@ -563,7 +575,7 @@ def checkForFinishedThreads(self): t.status = 'ended' def sync(self, expectedThreads=0): - 'Wait for all but expectedThreads async tasks to finish.' + 'Wait for all but expectedThreads async threads to finish.' while len(self.unfinishedThreads) > expectedThreads: self.checkForFinishedThreads() @@ -677,8 +689,10 @@ def findMatchingColumn(sheet, row, columns, func): status('%s matches for /%s/' % (matchingRowIndexes, regex.pattern)) - def exceptionCaught(self, status=True): + def exceptionCaught(self, exc=None, status=True): 'Maintain list of most recent errors and return most recent one.' + if isinstance(exc, ExpectedException): # already reported, don't log + return self.lastErrors.append(stacktrace()) if status: return self.status(self.lastErrors[-1][-1]) # last line of latest error @@ -692,7 +706,7 @@ def drawLeftStatus(self, scr, vs): attr = colors[options.color_status] _clipdraw(scr, self.windowHeight-1, 0, lstatus, attr, self.windowWidth) except Exception as e: - self.exceptionCaught() + self.exceptionCaught(e) def drawRightStatus(self, scr, vs): 'Draw right side of status bar.' @@ -706,7 +720,7 @@ def drawRightStatus(self, scr, vs): attr = colors[color] _clipdraw(scr, self.windowHeight-1, rightx, rstatus, attr, len(rstatus)) except Exception as e: - self.exceptionCaught() + self.exceptionCaught(e) curses.doupdate() @@ -753,7 +767,7 @@ def run(self, scr): try: sheet.draw(scr) except Exception as e: - self.exceptionCaught() + self.exceptionCaught(e) self.drawLeftStatus(scr, sheet) self.drawRightStatus(scr, sheet) # visible during this getkeystroke @@ -806,8 +820,8 @@ def run(self, scr): self.callHook('predraw') try: sheet.checkCursor() - except Exception: - exceptionCaught() + except Exception as e: + exceptionCaught(e) def replace(self, vs): 'Replace top sheet with the given sheet `vs`.' @@ -853,6 +867,30 @@ def __getitem__(self, k): def __setitem__(self, k, v): setattr(self.obj, k, v) +class CompleteExpr: + def __init__(self, sheet=None): + self.sheet = sheet + + def __call__(self, val, state): + i = len(val)-1 + while val[i:].isidentifier() and i >= 0: + i -= 1 + + if i < 0: + base = '' + partial = val + elif val[i] == '.': # no completion of attributes + return None + else: + base = val[:i+1] + partial = val[i+1:] + + varnames = [] + varnames.extend(sorted((base+col.name) for col in self.sheet.columns if col.name.startswith(partial))) + varnames.extend(sorted((base+x) for x in globals() if x.startswith(partial))) + return varnames[state%len(varnames)] + + class Colorizer: def __init__(self, colorizerType, precedence, colorfunc): self.type = colorizerType @@ -890,8 +928,8 @@ def __init__(self, name, **kwargs): self.rowLayout = {} # [rowidx] -> y self.visibleColLayout = {} # [vcolidx] -> (x, w) - # all columns in display order - self.columns = kwargs.get('columns') or [copy(c) for c in self.columns] # list of Column objects + # list of all columns in display order + self.columns = kwargs.get('columns') or [copy(c) for c in self.columns] or [Column('')] self.recalc() # commands specific to this sheet @@ -906,7 +944,7 @@ def __init__(self, name, **kwargs): # for progress bar self.progresses = [] # list of Progress objects - # track all async tasks from sheet + # track all async threads from sheet self.currentThreads = [] self._colorizers = {'row': [], 'col': [], 'hdr': [], 'cell': []} @@ -964,6 +1002,7 @@ def addRow(self, row, index=None): self.rows.append(row) else: self.rows.insert(index, row) + return row def searchColumnNameRegex(self, colregex, moveCursor=False): 'Select visible column matching `colregex`, if found.' @@ -1042,6 +1081,9 @@ def __repr__(self): def evalexpr(self, expr, row): return eval(expr, getGlobals(), LazyMapRow(self, row)) + def inputExpr(self, prompt, *args, **kwargs): + return input(prompt, "expr", *args, completer=CompleteExpr(self), **kwargs) + def getCommand(self, keystrokes, default=None): k = keystrokes cmd = None @@ -1069,13 +1111,13 @@ def exec_command(self, cmd, args='', vdglobals=None, keystrokes=None): except EscapeException as e: # user aborted self.vd.status('aborted') escaped = True - except Exception: - err = self.vd.exceptionCaught() + except Exception as e: + err = self.vd.exceptionCaught(e) try: self.vd.callHook('postexec', self.vd.sheets[0] if self.vd.sheets else None, escaped, err) except Exception: - self.vd.exceptionCaught() + self.vd.exceptionCaught(e) self.vd.refresh() return escaped @@ -1245,8 +1287,11 @@ def unselectByIdx(self, rowIdxs): def gatherBy(self, func): 'Generate only rows for which the given func returns True.' for r in Progress(self.rows): - if func(r): - yield r + try: + if func(r): + yield r + except Exception: + pass def orderBy(self, *cols, **kwargs): self.rows.sort(key=lambda r,cols=cols: tuple(c.getTypedValue(r) for c in cols), **kwargs) @@ -1331,14 +1376,21 @@ def toggleKeyColumn(self, colidx): moveListItem(self.columns, colidx, self.nKeys) return 0 + def rowkey(self, row): + 'returns a tuple of the key for the given row' + return tuple(c.getValue(row) for c in self.keyCols) + def moveToNextRow(self, func, reverse=False): 'Move cursor to next (prev if reverse) row for which func returns True. Returns False if no row meets the criteria.' rng = range(self.cursorRowIndex-1, -1, -1) if reverse else range(self.cursorRowIndex+1, self.nRows) for i in rng: - if func(self.rows[i]): - self.cursorRowIndex = i - return True + try: + if func(self.rows[i]): + self.cursorRowIndex = i + return True + except Exception: + pass return False @@ -1400,8 +1452,11 @@ def calcColLayout(self): for vcolidx in range(0, self.nVisibleCols): col = self.visibleCols[vcolidx] if col.width is None and self.visibleRows: + # handle delayed column width-finding col.width = col.getMaxWidth(self.visibleRows)+minColWidth - width = col.width if col.width is not None else col.getMaxWidth(self.visibleRows) # handle delayed column width-finding + if vcolidx != self.nVisibleCols-1: # let last column fill up the max width + col.width = min(col.width, options.default_width) + width = col.width if col.width is not None else options.default_width if col in self.keyCols or vcolidx >= self.leftVisibleColIndex: # visible columns self.visibleColLayout[vcolidx] = [x, min(width, winWidth-x)] x += width+sepColWidth @@ -1642,7 +1697,7 @@ def getTypedValue(self, row): try: return self.type(self.getValue(row)) except Exception as e: -# exceptionCaught(status=False) +# exceptionCaught(e, status=False) return self.type() def getValue(self, row): @@ -1707,12 +1762,12 @@ def getDisplayValue(self, row): return self.getCell(row).display def setValue(self, row, value): - if not self.setter: - error('column cannot be changed') + 'Set our column value for given row to `value`.' + self.setter or error(self.name+' column cannot be changed') self.setter(self, row, value) def setValues(self, rows, value): - 'Set given rows to `value`.' + 'Set our column value for given list of rows to `value`.' value = self.type(value) for r in rows: self.setValue(r, value) @@ -1895,7 +1950,7 @@ class TextSheet(Sheet): 'Displays any iterable source, with linewrap if wrap set in init kwargs or options.' rowtype = 'lines' commands = [ - Command('w', 'sheet.wrap = not getattr(sheet, "wrap", options.wrap); status("text%s wrapped" % (" NOT" if wrap else "")); reload()', 'toggle text wrap for this sheet') + Command('v', 'sheet.wrap = not getattr(sheet, "wrap", options.wrap); status("text%s wrapped" % ("" if wrap else " NOT")); reload()', 'toggle text wrap for this sheet') ] filetype = 'txt' @@ -1918,29 +1973,36 @@ def reload(self): class ColumnsSheet(Sheet): rowtype = 'columns' class ValueColumn(Column): + 'passthrough to the value on the source cursorRow' def calcValue(self, srcCol): - return srcCol.getDisplayValue(self.sheet.source.cursorRow) + return srcCol.getDisplayValue(srcCol.sheet.cursorRow) def setValue(self, srcCol, val): srcCol.setValue(self.sheet.source.cursorRow, val) columns = [ + ColumnAttr('sheet'), ColumnAttr('name'), ColumnAttr('width', type=int), ColumnEnum('type', globals(), default=anytype), ColumnAttr('fmtstr'), ValueColumn('value') ] - nKeys = 1 + nKeys = 2 colorizers = [ - Colorizer('row', 7, lambda self,c,r,v: options.color_key_col if r in self.source.keyCols else None), - Colorizer('row', 8, lambda self,c,r,v: 'underline' if self.source.nKeys > 0 and r is self.source.keyCols[-1] else None) + Colorizer('row', 7, lambda self,c,r,v: options.color_key_col if r in r.sheet.keyCols else None), ] commands = [] def reload(self): - self.rows = self.source.columns - self.cursorRowIndex = self.source.cursorColIndex - + if isinstance(self.source, Sheet): + self.rows = self.source.columns + self.cursorRowIndex = self.source.cursorColIndex + self.columns[0].width = 0 # hide 'sheet' column if only one sheet + else: # lists of Columns + self.rows = [] + for src in self.source: + if src is not self: + self.rows.extend(src.columns) class SheetsSheet(Sheet): rowtype = 'sheets' @@ -1990,7 +2052,7 @@ def reload(self): class OptionsSheet(Sheet): rowtype = 'options' commands = [ - Command(ENTER, 'source[cursorRow[0]] = editCell(1)', 'edit option'), + Command(ENTER, 'editOption(cursorRow)', 'edit option'), Command('e', ENTER) ] columns = [ColumnItem('option', 0), @@ -2000,6 +2062,12 @@ class OptionsSheet(Sheet): colorizers = [] nKeys = 1 + def editOption(self, row): + if isinstance(row[2], bool): + self.source[row[0]] = not row[1] + else: + self.source[row[0]] = self.editCell(1) + def reload(self): self.rows = list(self.source._opts.values()) @@ -2100,15 +2168,38 @@ def delchar(s, i, remove=1): 'Delete `remove` characters from str `s` beginning at position `i`.' return s if i < 0 else s[:i] + s[i+remove:] - def complete(v, comps, cidx): - 'Complete keystroke `v` based on list `comps` of completions.' - if comps: - for i in range(cidx, cidx + len(comps)): - i %= len(comps) - if comps[i].startswith(v): - return comps[i] - # beep - return v + class CompleteState: + def __init__(self, completer_func): + self.comps_idx = -1 + self.completer_func = completer_func + self.former_i = None + self.just_completed = False + + def complete(self, v, i, state_incr): + self.just_completed = True + self.comps_idx += state_incr + + if self.former_i is None: + self.former_i = i + try: + r = self.completer_func(v[:self.former_i], self.comps_idx) + except Exception as e: + # beep/flash; how to report exception? + return v, i + + if not r: + # beep/flash to indicate no matches? + return v, i + + v = r + v[i:] + return v, len(v) + + def reset(self): + if self.just_completed: + self.just_completed = False + else: + self.former_i = None + self.comps_idx = -1 class HistoryState: def __init__(self, history): @@ -2139,13 +2230,20 @@ def down(self, v, i): return v, i history_state = HistoryState(history) + complete_state = CompleteState(completer) insert_mode = True first_action = True v = str(value) # value under edit i = 0 # index into v - comps_idx = -1 left_truncchar = right_truncchar = truncchar + def rfind_nonword(s, a, b): + while not s[b].isalnum() and b >= a: # first skip non-word chars + b -= 1 + while s[b].isalnum() and b >= a: + b -= 1 + return b + while True: if display: dispval = clean_printable(v) @@ -2180,8 +2278,8 @@ def down(self, v, i): elif ch == '^E' or ch == 'KEY_END': i = len(v) elif ch == '^F' or ch == 'KEY_RIGHT': i += 1 elif ch in ('^H', 'KEY_BACKSPACE', '^?'): i -= 1; v = delchar(v, i) - elif ch == '^I': comps_idx += 1; v = completer(v[:i], comps_idx) or v - elif ch == 'KEY_BTAB': comps_idx -= 1; v = completer(v[:i], comps_idx) or v + elif ch == '^I': v, i = complete_state.complete(v, i, +1) + elif ch == 'KEY_BTAB': v, i = complete_state.complete(v, i, -1) elif ch == ENTER: break elif ch == '^K': v = v[:i] # ^Kill to end-of-line elif ch == '^O': v = launchExternalEditor(v) @@ -2189,6 +2287,7 @@ def down(self, v, i): elif ch == '^T': v = delchar(splice(v, i-2, v[i-1]), i) # swap chars elif ch == '^U': v = v[i:]; i = 0 # clear to beginning elif ch == '^V': v = splice(v, i, until_get_wch()); i += 1 # literal character + elif ch == '^W': j = rfind_nonword(v, 0, i-1); v = v[:j+1] + v[i:]; i = j+1 elif ch == '^Z': v = suspend() elif history and ch == 'KEY_UP': v, i = history_state.up(v, i) elif history and ch == 'KEY_DOWN': v, i = history_state.down(v, i) @@ -2206,6 +2305,7 @@ def down(self, v, i): if i < 0: i = 0 if i > len(v): i = len(v) first_action = False + complete_state.reset() return v @@ -2216,10 +2316,16 @@ def __init__(self): self.color_attrs = {} def setup(self): + if options.use_default_colors: + curses.use_default_colors() + default_bg = -1 + else: + default_bg = curses.COLOR_BLACK + self.color_attrs['black'] = curses.color_pair(0) for c in range(0, options.force_256_colors and 256 or curses.COLORS): - curses.init_pair(c+1, c, curses.COLOR_BLACK) + curses.init_pair(c+1, c, default_bg) self.color_attrs[str(c)] = curses.color_pair(c+1) for c in 'red green yellow blue magenta cyan white'.split(): @@ -2259,8 +2365,6 @@ def setupcolors(stdscr, f, *args): curses.mouseinterval(0) # very snappy but does not allow for [multi]click curses.mouseEvents = {} - curses.use_default_colors() - for k in dir(curses): if k.startswith('BUTTON') or k == 'REPORT_MOUSE_POSITION': curses.mouseEvents[getattr(curses, k)] = k diff --git a/www/design/async.md b/www/design/async.md index 67086c581..7ed2b0df0 100644 --- a/www/design/async.md +++ b/www/design/async.md @@ -28,10 +28,10 @@ This will only rarely be useful. # Threads Sheet (`^T`) -All threads (active, aborted, and completed) are added to `VisiData.threads`, which can be viewed as the Task Sheets via `^T`. -Threads which take less than `min_task_time_s` (hardcoded in `async.py` to 10ms) are removed, to reduce clutter. +All threads (active, aborted, and completed) are added to `VisiData.threads`, which can be viewed as the ThreadsSheet via `^T`. +Threads which take less than `min_thread_time_s` (hardcoded in `async.py` to 10ms) are removed, to reduce clutter. -- Press `ENTER` (on the Threads Sheet) on a thread to view its performance profile (if `options.profile_tasks` was True when the thread started). +- Press `ENTER` (on the Threads Sheet) on a thread to view its performance profile (if `options.profile_threads` was True when the thread started). - Press `^_` (anywhere) to toggle profiling of the main thread. ## Profiling diff --git a/www/design/commands.md b/www/design/commands.md index ae5a0a619..e90cc4d3c 100644 --- a/www/design/commands.md +++ b/www/design/commands.md @@ -138,7 +138,7 @@ But they are not, in part because they are assumed to be used infrequently (and - ^U - paUse/resUme replay - ^P - Previous status messages (like nethack) - ^Q - force quit -- ^T - Tasks sheet (should this be Shift-T?) +- ^T - Threads sheet - ^X - eval python eXpression; g^X exec python expression (for import) - ^Y - push object sheet of this row; z^Y push object sheet of this cell value diff --git a/www/design/graphics.md b/www/design/graphics.md new file mode 100644 index 000000000..df42b3f53 --- /dev/null +++ b/www/design/graphics.md @@ -0,0 +1,136 @@ +# Terminal Graphics in VisiData + +VisiData can display low-resolution terminal graphics with a reasonable amount of user interactivity. + +The current implementation uses braille Unicode characters (inspired by [asciimoo/drawille](https://github.com/asciimoo/drawille)). [Unicode blocks](https://en.wikipedia.org/wiki/Block_Elements) or the [sixel protocol](https://en.wikipedia.org/wiki/Sixel) may be supported in the future. + +## Class hierarchy + +- `Sheet`: the drawable context base class (part of core vdtui.py) +- `Plotter`: pixel-addressable display of entire terminal with (x,y) integer pixel coordinates +- `Canvas`: zoomable/scrollable virtual canvas with (x,y) coordinates in arbitrary units +- `InvertedCanvas`: a Canvas with inverted y-axis +- `Graph`: an InvertedCanvas with axis labels, a legend, and gridlines + +### Summary + +- The async `Graph.reload()` iterates over the given `sourceRows` (from its `source` Sheet) and calls `Canvas.polyline()` to indicate what to render. +- `Canvas.refresh()` triggers an async `Canvas.render()`, which iterates over the polylines and labels and calls the `Plotter.plot*` methods. +- The `VisiData.run()` loop calls `Plotter.draw()`, which determines the characters and colors to represent the pixels. + +### class `Plotter` + +A `Plotter` is a [`Sheet`](/design/sheet) with a pixel-addressable drawing surface that covers the entire terminal (minus the status line). Pixels and labels are plotted at exact locations in the terminal window, and must be recalculated after any zoomlevel change or terminal resizing. + +`Plotter.draw(scr)` is called multiple times per second to update the screen, and chooses a curses attribute for each pixel. +By default, the most common attr is chosen for each pixel, but if `options.disp_pixel_random` is set, an attr will be randomly chosen from the naturally weighted set of attrs (this may cause blocks of pixels to flicker between their possible attrs). +If an attr is in the `Canvas.hiddenAttrs` set, then it is not considered for display at all (and its rows will be ignored during selection). + +All Plotter coordinates must be integer numbers of pixels. +[For performance reasons, they are presumed to already be integers, to save unnecessary calls to `round()`.] +Methods which plot multiple pixels on the canvas should be careful to gauge the display correctly; simply calling `round()` on each calculated float coordinate will work but can cause display artifacts. + +#### `Plotter` methods + +For Plotter methods, `x` and `y` must be integers, where `0 <= x < plotwidth`, and `0 <= y < plotheight`. `(0,0)` is in the upper-left corner of the terminal window. + +Pixels can be plotted directly onto a Plotter with these methods: + +- `Plotter.plotpixel(x, y, attr, row=None)` +- `Plotter.plotline(x1, y1, x2, y2, attr, row=None)` +- `Plotter.plotlabel(x, y, text, attr)` + +`attr` is a [curses attribute](/design/color), and `row` is the object associated with the pixel. + +The above `plot*` methods append the `row` to `Plotter.pixels[y][x][attr]`. + +These properties and methods are also available: + +- `Plotter.plotwidth` is the width of the terminal, in pixels. +- `Plotter.plotheight` is the height of the terminal, in pixels. +- `Plotter.rowsWithin(bbox)` generates the rows plotted within the given region. +- `Plotter.hideAttr(attr, hide=True)` adds attr to `hiddenAttrs` if `hide`, and removes it otherwise. +- `Plotter.refresh()` is called whenever the screen size changes, and should also be invoked whenever new content is added. + +`rowsWithin` takes a `Box` object (described below). The `Box` class is otherwise unused by the Plotter. + +### class `Canvas` + +A **`Canvas`** is a `Plotter` with a virtual surface on which lines and labels can be rendered in arbitrary units. + +The onscreen portion (the area within the visible bounds) is scaled and rendered onto the `Plotter`, with the minimum coordinates in the upper-left [same orientation as `Plotter`]. + +The [`Canvas` user interface](/howto/graph#commands) supports zoom, scroll, cursor definition, and selection of the underlying rows. The `source` attribute should be the Sheet which owns the plotted `row` objects. + +A call to `Canvas.refresh()` will trigger `Canvas.render()`, which is decorated with `@async` as it may take a perceptible amount of time for larger datasets. Any active `render` threads are cancelled first. + +#### `Box` and `Point` helper classes + +While the Plotter API requires literal integer values for `x`/`y` and `width`/`height` parameters, `Canvas` methods generally take float values contained in either `Box` or `Point` classes. + +##### `Point` + +`Point` is simply a container for an `(x,y)` coordinate (passed to the constructor). The individual components are stored as `.x` and `.y`, and the computed `.xy` property will return `(x,y)` as a simple tuple. `Point` can also stringify itself reasonably. + +##### `Box` + +`Box` is effectively a rectangle stretching over some area of the canvas. The constructor takes `(x,y,w,h)`, but a `Box` can also be constructed using the `BoundingBox(x1,y1,x2,y2)` helper. [Note that in the BoundingBox case, the order of the individual points is not guaranteed; the individual coordinates may be swapped for convenience.] + +`Box` has these members and properties: + +- `xmin` and `ymin`: the minimum coordinates of the area. +- `xmax` and `ymax`: the maximum coordinates of the area. +- `xcenter` and `ycenter`: the central coordinates of the area. +- `w` and `h`: the width and height of the area. +- `xymin`: returns `Point(xmin,ymin)`. +- `center`: returns `Point(xcenter,ycenter)`. +- `contains(x, y)`: returns True if `(x,y)` is inside the bounding box. + +#### `Canvas` methods + +- `Canvas.polyline(vertexes, attr, row=None)` adds a multi-segment line from the list of (x,y) `vertexes`. One vertex draws a point; two vertexes draws a single line. Note that the vertexes are *not* Point objects (unlike parameters for other methods). +- `Canvas.label(xy, text, attr, row=None)` adds `text` at `xy` (Point in canvas units). +- `Canvas.fixPoint(xyplotter, xycanvas)` sets the position of the `visibleBox` so that `xycanvas` (Point in Canvas units) is plotted at `xyplotter` (Point in Plotter units). +- `Canvas.zoomTo(bbox)` sets the visible bounds so the given canvas coordinates will fill the entire Plotter area. `aspectRatio` will still be obeyed. +- `Canvas.keyattr(key)` returns the `attr` for the given `key`, assigning a new color from `options.plot_colors` if `key` has not been seen before. These keys are plotted as legends on the upper right corner of the canvas. The last color is given out for all remaining keys and is labeled "other". +- `Canvas.resetBounds()` needs to be called after some or all points have been rendered, but before anything can be plotted. It initializes the width and height of the canvas, visible area, and/or cursor. +- `Canvas.reset()` clears the canvas in preparation for `reload()`. + +#### `Canvas` properties + +- `Canvas.canvasBox` reflects the bounds of the entire canvas. +- `Canvas.visibleBox` defines the onscreen canvas area. +- `Canvas.cursorBox` defines the cursor region in canvas coordinates. +- `Canvas.zoomlevel` is a settable property, which sets the `visibleBox` size accordingly. `zoomlevel` of 1.0 makes the entire canvas visible. Does not change the position of the `visibleBox` (see `Canvas.fixPoint`). +- `Canvas.aspectRatio`, if set, maintains a proportional width and height of the `visibleBox` (considering also `plotwidth`/`plotheight`). `aspectRatio` of 1.0 should be square. +- `Canvas.canvasCharWidth` and `Canvas.canvasCharHeight` is the width and height of one terminal character, in canvas units. + +These properties reserve an area of the Plotter that is outside the visibleBox: +- `Canvas.leftMarginPixels` +- `Canvas.rightMarginPixels` +- `Canvas.topMarginPixels` +- `Canvas.bottomMarginPixels` + +During a [mouse event](/design/commands#mouse), these properties indicate the mouse position for the current mouse event: + +- `Canvas.canvasMouse`: a Point in canvas coordinates +- `Plotter.plotterMouse`: a Point in plotter (pixel) coordinates +- `Sheet.mouseX` and `Sheet.mouseY`: individual values in curses (character) coordinates + +### class `InvertedCanvas` + +An `InvertedCanvas` is a `Canvas` with a few internal methods overridden, such that the Y axis is inverted. For an `InvertedCanvas`, the minimum coordinates are in the lower-left. + +`InvertedCanvas` has not much else of interest. It should be completely interchangeable with `Canvas`. + +### class `Graph` + +A `Graph` is an `InvertedCanvas` with axis labels and/or gridlines. + +- `Graph.__init__(name, sheet, rows, xcols, ycols)` constructor + - `sheet` is the `source` Sheet. + - `rows` is a list of the rows to iterate over (from the given `source`). + - `xcols` is a list of key columns forming the x-axis and color keys. + - `ycols` is a list of numeric columns to be plotted on the y-axs. + +--- diff --git a/www/devotees.gpg.key b/www/devotees.gpg.key new file mode 100644 index 000000000..af0c005c6 --- /dev/null +++ b/www/devotees.gpg.key @@ -0,0 +1,65 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- +Version: GnuPG v1 + +mQINBFo4clMBEADGcX0dX2gWsd518aHKuD9sNTaLpq2KNjD7CLBugj5d2G/4h1Ew +6suEFLBsaFH8/zFWUfZ2zvxN3M5wi0FNEvHh7pi3oxDOETI2KmFyy4NyxVUfBEmh +PbD3i/GOi7MxSpKkhl7q38P5MWN9WPYc5gZpfyIX7aeHtFog3u2gebcfckD0tGAJ +6PQZiv7ZvuJCSs1Lp7BSXjSgClUD+ng/DP6QqfcBuNdna2H33DP+m69Ktgg1RWOg +dgSNJi0uvlpjkpqVS4UiKSh/4AFwvQzEYiCkuNm+JDlwrJfIfsDK+ftjD1Fa7mcZ +aIaOlg1qi0KarSydumE6WItXlCsTX3DO2Jn8GvN7sBT79/zQfxBDd6603Wn2MgjO +z3b8dxrjGjLslJbfyxvxLjung5N/wVk6KHDkuIHW4esAknJInnrBo0dSaV2lIm+G +MLDnYEEa/L65erfZEA7gUEOHRyVHl1K50RRC1NNG4M0OBI8AhQOLLtSiK8nmpiU4 +ETgaftM1YYcDqmK+6QQGuZqRRaPJH5ENS5RbWTkd5AYhnxjuSun+Oa3IdmYH1C+6 +wndskRoIJ0bHeTvZPX3WS3606ZqriN3/AwYbk8o5XKQHGMxW8llK0rn0sjoVpT1N +unVPtKxFOcOoixMhDbYhJ8WA4h8tq0n+3ylliwjENcSHJrzSNpwT82ODywARAQAB +tCpBbmphIEJvc2tvdmljIChLZWZhbGEpIDxhbmphQHZpc2lkYXRhLm9yZz6JAj8E +EwEIACkCGwMFCQWjmoAECwgHAwUVCgkICwUWAgMBAAIeAQIXgAUCWjhzIgIZAQAK +CRAcq5rZl14K+vWVD/wKOoK6acpzT79RbG9fCt9ARVRaNQjVL47vf0SQkHBbi5kk +r/6+Y6JNiclaLRNh7xorDodh2H8Z4YLdFKlJ44p22i96owwn0XEiFDZJPeJMBDnf +N5Qb6081+bix0b4GqwYfJPSQ6DkfIiNNi4qAXd1VRCUll1UkVS+Yqr/2jw93qfIw +ihx91nzw5Qp0gC/edfewA62oFUEpgFw2tm2+L4h4VcvLE6/ylqGiYKgbESk2wzgt +1DZIzx9ifn0tjlSvJZ2c3pNm1ZK1qD837QtsMtYwpQeGqYh3AyHbM7FrVpMsT136 +Tm+CWWUSrjw5YAexnG7OMnUWokn9fjKJGYl8h0fu5EQS1S+G9Ix/alTrHCHXzVO7 +wCh1acrZIWCfgImRcfi+O8V0H1cjdo6cG2VIIgNh4zoQtUQ2rTHxV+DUWOScRjF4 +NrWFZHifPNnh+1enR5kUtRe61lktic9SxxvkQHcdfiKBm+xk9iQppmLbBJucqxx5 +ItRksMiAoszvTs4UkOepKBGyEISTCRm1+1fHGqf/39IaCrmlMqeyZPBPQanACj4S +PYnsSvIN9AbaM8dnAWtvEAlYMxq/gb1hrB5l3n9aPo3QW5s4ZHz+DBUcPwMw/H46 +BolFT9WEO755apqyVzngUZ31B5KIOsSEBLT0kVNVw0Q4UIdU3eOkXS/+Ewpb5rQu +QW5qYSBCb3Nrb3ZpYyAoS2VmYWxhKSA8YW5qYS5rZWZhbGFAZ21haWwuY29tPokC +PAQTAQgAJgUCWjhy/gIbAwUJBaOagAQLCAcDBRUKCQgLBRYCAwEAAh4BAheAAAoJ +EByrmtmXXgr6EbcP/jneuXhwAG6qAKaSDKg9Z8lyfkS0c+b482LjtBQCfJ1lZ3vy +q6GsMte96rsb/PN2S65jrjljfiFvSnLkcPKxV204jTff3NaarpHYc14hRZjLf1ir +FMzHPiCtoPXj/6GaNo5NJxPv5vQmTAMTjwVF3teXoHUGX4w5as+MTPWyD8wL4G1J +Y1uMBoJ0QX40Njk8Ve/jBEY84MKbvQhldxkeE/NF0a16UvtxRJcWmYRLQ01XvDnh +9FqEvNY3dSCsa+jqkcL5TzQN6eBz3mZKH61jpqlUfha4eacNYW1dagm4UjCjvh4K ++RSlUQ/HOvJ26Bqk0wyAmz/mrMWERljrZFDe2AafdlK3gVdXhhWh30q+zHjKTFtI +tlcwc+TwrKZ/KibPn2MqBBTsZNv3UqOt3DEPhHBnxPZlUP8t9azfXivxnTH2hzEQ +1uIPCRQb+7yVQxmKmYj2wFsBj8J4hAP18pyh+BwP+h4CJayqg2GIx2T5exqsLJ1/ +X/DVrtRImKHgBEnxmpW8NnD7Ia9GVFHbd1+2s8HaF5y9fef+Bf7V8ZV2+scQKYZ1 +EoG69bgrpBJRr9+dCQcXfvuSwLXWqObDPMHyRcbcMx0Madm+KPw+/+zz2GPsFE3n +hqPhn4IgZ0UKLULKnGADu2GSgxhAqZZOxDFnzm8FC1hMtHl9aSe0GsoXF+Z0uQIN +BFo4clMBEACmXvQ9utW38OcMyocXf5OjUktyDQVE7JTtn52ZqucBUtmYT1m7HqC9 +5Bh+IPQv02mxKs8Tk1IPVAqXkJdUdbl8ssd/Txj3ENN70tXGFyI6Bv5cvd6hyG8P +g5K/4dsZlk+eCYDGd4m1lTQkjVdCw5AH0rJHrAWUcer3/zPL6xTH+6wTNFZUFpYu +jn0/091FNw4a9norpZ/MIsluETPl9y7gotoOXsODnWhi7/ewe9VHppZiY4TLJAsh +FY1ByX/jq5ffB4r1pbdflG7mCwsNcV9YDgSgPXXLUxAks1+QYAbz+OwiI2j3Ccb5 +zgRgTPYoPTl2BAOt/TbUaiGj784KulgLWmpQzVDFUiVBQ2mItyzJkfLZdYAjHVwL +EH6kSZlIlzPrhHu7u5MkDshVVj4ylnnc0Djpakr0SqmBCXNXQl7oxB2/6z9zpLiR +WVgo6HHh7MDCSwUGEq9B6LeuTWlbNvmjRUByAGcy7XcX+pCZbe2l6ijeH6utWjS3 +dj2vvyb7pITQrqZYoKJWLHR4AeGQV+u98aY8lv1H8kSCELVdWyc4gc3iXxI9IJWU +BTiD8KYaH3k70ZxSqJP7A1nyPpw3yQ3ESkNugSiRpJTozg5S6BVaj33SqeEszcAZ +0HSuIUWoyhEJIzGRkEaZTyfpLzDmgJD3/jtIjjfRSFJqcurIGTClSwARAQABiQIl +BBgBCAAPBQJaOHJTAhsMBQkFo5qAAAoJEByrmtmXXgr6DfgP/10p3XHdoSMd/TbN +bM4m9Ul5FARPbj91o6ulNmR94/lfZv+2sI+Tg4jntp56WKVKlgWrCT/m+TjWO6Es +6babgeVxN6I5ZObiqI4OoJUFUs+76qVSY25JQGibG/uXLbGGubJv0tWf26RczWLv +1voa4LXsD30rd8PDyH/tofS4+KXtcAxbTQ7EUEHGhOabvKT2/kUFig3ybSktJIhx +RJMCdsdR8CM16p8wU2iQZ3ZtilkofXS2ET69KU7g4rLSeNDr5pH1+iknwRYa1YsA +X1wZ44kp78mSXkdJoUaZxYiccpBIRiSRH3F25Eve/gOWmzzspL4gDk51UHHWK6Cx +N/uvaaT7MlRgxQ1yjeUE6bhzVCitf8+UVhAEUdT8u5sWU+yzUWCsF1K/QfUJk4Sa +cmj7NSLt6UsSMSQBqHPgs05qh0Z5YD9IPf4j8lYaE/oGyOm+Ty3A4DHogU9D7Wgp +Cac7yD0U5ESjIxsRF0GMFP5hVSzKhDvyk2mabDHXO6qHVrxvvQHjILk7x1Rmb0hl +WogM40gAT7aucJuqpvfSQOiClgMeqyiCvN7ik4LES4iLkbeaTO8iok6V6B9LVjNZ +3yYBf4xG906dzzyf0a32NwhkZC3SfgW+jzZ0KEwiqmAYOK2ywXOrL2J2JHmE0+Lo +PBxOVAdKCIZRxzPjrOSwWv1sYFDU +=hEnj +-----END PGP PUBLIC KEY BLOCK----- diff --git a/www/docs.md b/www/docs.md index b55cdb6d0..33b2032f2 100644 --- a/www/docs.md +++ b/www/docs.md @@ -1,17 +1,19 @@ # VisiData documentation -* VisiData's [manpage](/man) contains a comprehensive overview of all global and sheet-specific commands. +* VisiData's [manpage](/man) contains a comprehensive quick reference of all global and sheet-specific commands. -* Our [system tests](/test) provide examples of how VisiData's atomic commands can be combined for complex workflows. +* The [system tests](/test) provide examples of how VisiData's atomic commands can be combined for complex workflows. -* Our [CONTRIBUTING.md](/contributing) contains information on how to contribute to VisiData's ongoing development and where we can be found for different types of support needs. +* [CONTRIBUTING.md](/contributing) contains information on how to contribute to VisiData's ongoing development. -* We have some recordings of VisiData [demonstrations](/videos). More are in the works, but please let us know if there are any features in particular you wish to learn more about. +* There are a few VisiData [demo videos](/videos). Please leave a note in any of the community spaces if there are any features in particular you want to know about. ## For developers -VisiData's architecture is converging on an internal design and API, but it is not quite there. For this reason, we have decided not to release the developer documentation yet. We will publish documentation on internals as components become interface complete. +The dream is for VisiData to be able to interact with data from any source or in any format. -However, if you wish to expand VisiData so that it can work with a new beloved data format, it is not difficult, and we would love to work with you and provide support and direction. +Creating a loader for a new data source is easy and usually takes only an hour or two. It's fun and rewarding to create a great terminal interface for your favorite data source. -Currently, VisiData supports tsv, csv, hdf5, piping from stdin, json, mbtiles, postgres, shp, sqlite, xlsx, xls, and zip. We would love to find devs interested in creating loaders for xml, pandas dataframes, html, and many others. +Currently, VisiData supports [tsv, csv, json, html, sqlite, hdf5, xlsx, xls, shp, and mbtiles file formats. We would love to find devs interested in creating interfaces for all other data sources. + +* [How to build a loader](/howto/dev/loaders) diff --git a/www/frontpage-body.html b/www/frontpage-body.html index 1167a17be..cb0401b9a 100644 --- a/www/frontpage-body.html +++ b/www/frontpage-body.html @@ -6,26 +6,30 @@

VisiData

-

v0.98.1

+

v0.99

Rapidly explore columnar data in the terminal

VisiData is free, open source, and can be installed in seconds on any Linux or MacOS system with Python3 installed. - Just open your terminal and type: + Just open your terminal and choose your installation method:

-
-

- pip3 - install - visidata -

-
+
- +
@@ -55,7 +59,7 @@

Universal

Extensible

-

Add new commands, views, data sources, and even entire terminal applications, with a minimum of code and a maximum of flexibility. +

Add new commands, views, data sources, and even entire applications, with a minimum of code and a maximum of flexibility.

diff --git a/www/help/index.md b/www/help/index.md new file mode 100644 index 000000000..7f4ea32d1 --- /dev/null +++ b/www/help/index.md @@ -0,0 +1,38 @@ +# Support {#support} + +*How do I install VisiData?* + +There are three options for installing visidata: + +- [pip3](/install#pip3) for users who wish to import visidata into their own code or integrate it into their python virtual environment +- [brew](/install#brew) on MacOS/X for reliable installation of application components (such as the manpage) +- [apt](/install#apt) on Linux distributions + +*Where can I learn how to use VisiData?* + +We have documentation in various levels of detail available at [visidata.org/docs](http://visidata.org/docs/) from [an overview of all commands](http://visidata.org/man/) to [workflow recipes](http://visidata.org/test). + +If you have a workflow which you do not see covered, please don't hestitate to [file an issue](https://github.com/saulpw/visidata/issues/new) or post a comment in any of our [community spaces](https://github.com/saulpw/visidata/blob/stable/CONTRIBUTING.md#community). Our documentation is an ongoing effort, and we wish to prioritise the writing of recipes around user needs. + +*I found a bug!* + +[Create a GitHub issue](https://github.com/saulpw/visidata/issues/new) if something doesn't appear to be working right. If you get an unexpected error, please include the full stack trace that you get with `^E` and the saved Commandlog (`^D`). + +*Where can I go if I have further questions or requests?* + +- [r/visidata](http://reddit.com/r/visidata) on reddit +- [#visidata](irc://freenode.net/#visidata) on [freenode IRC](https://webchat.freenode.net) +- [saul@visidata.org](mailto:saul@visidata.org) to discuss feature requests and extensions +- [anja@visidata.org](mailto:anja@visidata.org) to discuss documentation/tests or to request tutorials + + +## Troubleshooting + +*Whenever I try graphing, I get an empty chart* + +Oh dear. Let us try to get to the bottom of this. Please run through the following options and contact us with the answers if the issue persists. + +1. Is the terminal set to use 256 colors? Most terminals do support 256 colors, but may have a different default configuration. Try adding `TERM=xterm-256color` to your `~/.bashrc`. +2. Which terminal program are you using? If you are using a Mac, does the same thing happen in both iTerm and Mac Terminal? +3. What color theme is your terminal set to? If you try with another theme, does that produce the same result? +4. Have you modified any other VisiData options (e.g. in `.visidatarc`)? Particularly theme/display options. diff --git a/www/howto/dev/loaders.md b/www/howto/dev/loaders.md new file mode 100644 index 000000000..307341e2a --- /dev/null +++ b/www/howto/dev/loaders.md @@ -0,0 +1,171 @@ +# How to create a loader for VisiData + + +The process of designing a loader is: + +1. Create a Sheet subclass; +2. Collect the rows from the sources in reload(); +3. Enumerate the available columns; +4. Create sheet-specific commands to interact with the rows, columns, and cells. + +## 1. Create a Sheet subclass + +When VisiData tries to open a source with filetype `foo`, it tries to call `open_foo(path)`. `path` is a [Path](/api/Path) object of some kind. + +Sheet constructors should take a name as their first parameter, and pass their `**kwargs` along to the [Sheet superclass](/api/Sheet), which will use some of them and set the rest as attributes on the new instance. + +Besides the name, the main thing the constructor has to do is set its `source` for `reload()`: + +``` +def open_foo(p): + return FooSheet(p.name, source=p) + +class FooSheet(Sheet): + def __init__(self, name, **kwargs): + super().__init__(self, name, **kwargs) + self.gpg = generate_key() +``` + +## 2. Collect data into rows + +`reload()` is called when the Sheet is first pushed, and thereafter by the user with `^R`. + +Using the Sheet `source`, `reload` populates `rows`: + +``` +class FooSheet(Sheet): + rowtype = 'foobits' # rowdef: Foo + # If the constructor does not have to do anything special, it can be omitted entirely. + def reload(self): + self.rows = [] + for r in crack_foo(self.source): + self.addRow(r) +``` + +The `rowtype` is for display purposes only. It should be plural. +The expected internal structure of the rows on this sheet always be declared in a comment with the searchable tag `rowdef`. + +### making it async + +1. Add `@async` decorator on `reload`. This causes the method to be launched in a new thread. + +2. Append each row one at a time. + - Use `addRow` in general, even though it merely calls `self.rows.append(r)`. + - Do not use a list comprehension, so that rows will be available before everything is loaded. + - Do not assume the order of the rows will be the allocate rows and then fill them in. The user may change the order of the rows + +3. Wrap the iterator with [Progress](/api/Progress). This updates the progress percentage as it passes each element through. + +``` +class FooSheet(Sheet): + rowtype = 'foobits' # rowdef: Foo + @async + def reload(self): + self.rows = [] + for r in Progress(crack_foo(self.source)): + self.addRow(r) +``` + +## 3. Enumerate the columns + +Each `Column` provides a different view into the row. + +In general, set the `columns` class member to a list of `Column`s: + +``` +class FooSheet(Sheet): + rowtype = 'foobits' # rowdef: Foo + columns = [ + ColumnAttr('name'), + Column('bar', getter=lambda col,row: row.inside[2], + setter=lambda col,row,val: row.set_bar(val)), + Column('baz', type=int, getter=lambda col,row: row.inside[1]*100) + ] + ... +``` + +If the columns aren't known beforehand (e.g. they have to be discovered while parsing the data), then they can be added with `addColumn` during `reload()`. +If this mechanism is used, reload has to clear the existing columns list first, or every reload will add another full set of columns. + +### Column properties + +Columns have a few properties, all optional in the constructor except for `name`: + +* **`name`**: should be a valid Python identifier and unique among the column names on the sheet. Otherwise the column cannot be used in an expression. + +* **`type`**: values can be `str` (`~`), `int` (`#`), `float` (`%`), `date` (`@`), `currency` (`$`). It can also be `anytype`, which passes the original value through unmodified. + +* **`width`**: specifies the initial width for the column; `0` means hidden, `None` (default) means calculate on first draw. + +* **`fmtstr`**: format string for use with `type`. May be [strftime-format]() for `date`, or `new-style format` for other types. + +#### `getter` + +The `Column` constructor is usually passed a `getter` lambda, at least. +For complex getters, a Column subclass can override the base method `Column.calcValue` instead. +The getter is the essential functionality of a `Column`. + +The `getter` lambda is passed the column instance (`col`) and the `row`, and returns the value of the cell. +The sheet may be gotten from `col.sheet`; `row in col.sheet.rows` should always be true. + +The default getter returns the entire row. + + +#### `setter` + +The `Column` may also be given a `setter` lambda, which allows a row to be modified (e.g. by a command that uses the `Sheet.editCell` method). +The `setter` lambda is passed the column instance (`col`), the `row`, and the new `value` to be set. + +By default there is no `setter`, so the column is read-only. + +In a Column subclass, `Column.setValue(self, row, value)` may be overridden instead. + +### Builtin Columns + +There are several helpers for constructing `Column` objects: + +* `ColumnAttr(attrname, **kwargs)` gets/sets an attribute from the row object using `getattr`/`setattr`. + This is useful when the rows are Python objects. + +* `ColumnItem(colname, itemkey, **kwargs)` uses the builtin `getitem` and `setitem` on the row. + This is useful when the rows are Python mappings or sequences, like dicts, lists, and tuples. + +* `SubrowColumn(origcol, subrowidx, **kwargs)` delegates to the original column with some part of the row. +This is useful for rows which are a list of references to other rows, like with joined sheets. + +A couple recurring patterns: + +- columns from a list of names: `[ColumnItem(name, i) for i, name in enumerate(colnames)]` +- columns from the first sample row, when rows are dicts: `[ColumnItem(k) for k in self.rows[0]]` + +## 4. Add Sheet-specific Commands + +``` +class FooSheet(Sheet): + rowtype = 'foobits' # rowdef: Foo + commands = [ + Command('b', 'cursorRow.set_bar(0)', 'reset bar to 0', 'reset-bar') + ] +``` + +A reasonably intuitive and mnemonic default keybinding should be chosen. +The [`execstr` and `helpstr`](/api/Command) +The longname allows the command to be rebound by a more descriptive name, and for the command to be redefined for other contexts (so all keys bound to that command will be redefined also). + +# Building a loader for a URL schemetype + +When VisiData tries to open a URL with schemetype of `foo` (i.e. starting with `foo://`), it calls `openurl_foo(urlpath, filetype)`. `urlpath` is a [UrlPath](/api/Path#UrlPath) object, with attributes for each of the elements of the parsed URL. + +`openurl_foo` should return a Sheet or call `error()`. +If the URL indicates a particular type of Sheet (like `magnet://`), then it should construct that Sheet itself. +If the URL is just a means to get to another filetype, then it can call `openSource` with a Path-like object that knows how to fetch the URL: + +``` +def openurl_foo(p, filetype=None): + return openSource(FooPath(p.url), filetype=filetype) +``` + +# Tests + +- `^R` reload +- try with large dataset, make sure it stays responsive and updates progress diff --git a/www/install/index.md b/www/install/index.md new file mode 100644 index 000000000..55553bc9f --- /dev/null +++ b/www/install/index.md @@ -0,0 +1,63 @@ +# Installation + +There are three options for installing visidata: + +- [pip3](/install#pip3) for users who wish to import visidata into their own code or wish to integrate it into a python virtual environment +- [brew](/install#brew) on MacOS for reliable installation of application components (such as the manpage) +- [apt](/install#apt) on Linux distributions + +## Install via pip3 + +This is the best installation method for users who wish to take advantage of VisiData in their own code, or integrate it into a Python3 virtual environment. + +To install VisiData, with loaders for the most common data file formats (including csv, tsv, fixed-width text, json, sqlite, http, html and xls): + +``` +$ pip3 install visidata +``` + +To install VisiData, plus external dependencies for all available loaders: + +``` +pip3 install "visidata[full]" +``` + +## Install via brew + +Ideal for MacOS users who primarily want to engage with VisiData as an application. This is currently the most reliable way to install VisiData's manpage on MacOS. + +``` +brew install saulpw/vd/visidata +``` + +Further instructions available [here](https://github.com/saulpw/homebrew-vd). + +## Install via apt + +Packaged for Linux users who do not wish to wrangle with PyPi or python3-pip. + +Currently, VisiData is undergoing review for integration into the main Debian repository. Until then it is available in our [Debian repo](https://github.com/saulpw/deb-vd). + +Grab our public key + +``` +wget http://visidata.org/devotees.gpg.key +apt-key add devotees.gpg.key +``` + +Add our repository to apt's search list + +``` +sudo apt-get install apt-transport-https +sudo vim /etc/apt/sources.list + deb[arch=amd64] https://raw.githubusercontent.com/saulpw/deb-vd/master sid main +sudo apt-get update +``` + +You can then install VisiData by typing: + +``` +sudo apt-get install visidata +``` + +Further instructions are available [here](https://github.com/saulpw/deb-vd). diff --git a/www/main.css b/www/main.css index 8c810cb99..9a429eb75 100644 --- a/www/main.css +++ b/www/main.css @@ -26,6 +26,12 @@ body { font-size: medium; } +em { + font-style: normal; + font-weight: bold; + font-size: 1.25em; +} + h1 { /* Correct the font size and margin on `h1` elements */ @@ -68,6 +74,7 @@ strong { /* --- Bullets and Lists --- */ li { display: list-item; + list-style-type: disc; margin: 0px auto; text-align: -webkit-match-parent; } @@ -267,14 +274,11 @@ pre > code { } code { - background-color: #f4f4f4; + background-color: #f0e4ff; box-shadow: inset 0px 0px 2px #bbb; font-family: Inconsolata, monospace; /* Correct the inheritance and scaling of font size in all browsers */ font-size: 1em; /* Correct the odd `em` font sizing in all browsers */ -/* padding-left: 0.3em; - padding-right: 0.3em; - padding-top: 0.1em; - padding-bottom: 0.1em; */ + padding: 0.25em; } pre, pre > code { @@ -380,7 +384,7 @@ footer { align-items: flex-start; background: #e0e0e0; - padding: 30px 0 30px 30px; + padding: 20px 0 20px 20px; } .signup { @@ -395,9 +399,30 @@ footer { flex-direction: column; flex-wrap: wrap; max-width: 400px; - padding: 30px; margin: auto; } + + .install-btns { + background-color: #8764c8; + border-radius: 3px; + font-size: 1em; + opacity: 0.5; + color: #d6cae7; + margin: auto; + text-align: center; + padding: 8px 15px; + transition: .2s linear; + display: inline-block; + } + + .install{ + text-align: center; + } + +.install-btns:hover { + color: #fff; + opacity: 1.0; +} .hero-image { flex: 0 1 100%; @@ -433,9 +458,8 @@ footer { .hero-cta-container { flex: 0 1 100%; + padding: 0.25em; background: #303B40; - box-shadow: inset 0 0 10px #000000; - border-radius: 10px; } .cta-code { @@ -455,7 +479,7 @@ footer { .feature { flex: 0 1 33%; - padding: 50px 0 20px 0; + padding: 15px 0 20px 0; display: flex; flex-direction: column; text-align: justify; @@ -469,7 +493,7 @@ footer { .feature-headline {} -.feature-description { +.feature-description > p { margin: 0 30px 0 30px; text-align: center; } diff --git a/www/news/news.tsv b/www/news/news.tsv index cca5b7f5d..73d7f281c 100644 --- a/www/news/news.tsv +++ b/www/news/news.tsv @@ -1,4 +1,5 @@ Date Version Version_href Info Info_href +2017-12-04 0.98.1 https://github.com/saulpw/visidata/releases/tag/v0.98.1 optional dependencies http://visidata.org/news/v0.98.1 2017-11-23 0.98 https://github.com/saulpw/visidata/releases/tag/v0.98 maps and graphs http://visidata.org/news/v0.98 2017-10-29 0.97.1 https://github.com/saulpw/visidata/releases/tag/v0.97.1 bugfixes http://visidata.org/news/v0.97.1 2017-10-06 0.97 https://github.com/saulpw/visidata/releases/tag/v0.97 manpage, replay http://visidata.org/news/v0.97 diff --git a/www/news/v0.98.1.md b/www/news/v0.98.1.md new file mode 100644 index 000000000..415e4d33e --- /dev/null +++ b/www/news/v0.98.1.md @@ -0,0 +1,10 @@ +v0.98.1 is a patch release that fixes a couple of minor bugs. The primary change in this release, however, is that the 'visidata' package on PyPI no longer includes all of the loaders' dependencies by default. + +This is a minor hassle for first-time users if they want to use certain formats, but since the goal is eventually supporting every possible data format, installing all dependencies for all users by default is not tenable. The base VisiData already includes support for tsv, csv, fixed width, sqlite, graphs and more. Most users will only need one or two additional dependencies. + +To install VisiData with all dependencies for all loaders: + + pip3 install "visidata[full]" + +Of course, you can install just the dependencies you need. See ["SUPPORTED SOURCES"](http://visidata.org/man/#loaders) in the manpage for which packages to install. + diff --git a/www/template.html b/www/template.html index 34e0aa98c..074ab9ce2 100644 --- a/www/template.html +++ b/www/template.html @@ -27,7 +27,9 @@

home

about

+

install

documentation

+

support

release history