Merge pull request #99 from saulpw/develop

v0.99 release

CHANGELOG.md

@@ -1,5 +1,24 @@
 # VisiData version history
 
+# v0.99 (2017-12-22)
+
+- tab completion for filename and python expr
+- `v` now 'visibility toggle' (moved from `w`)
+- `^W` to erase a word in the line editor
+- `gC`
+- `--version` (thanks for the suggestion, @jsvine)
+- `options.use_default_colors`
+- `median` aggregator
+- .html loads tables (requires lxml)
+  - simple http works (requires requests)
+- json save
+- json incremental load
+- [cmdlog] use rowkey if available instead of row number; options.rowkey_prefix
+- [cmdlog] only set row/col when relevant
+- [vdtui] task renamed to thread
+- /howto/dev/loader
+- /design/graphics
+
 # v0.98.1 (2017-12-04)
 
 - [packaging]

CONTRIBUTING.md

@@ -17,11 +17,11 @@ In addition, the innermost core file, `vdtui.py`, is a single-file stand-alone l
 
 ### Feature requests or enhancements
 
-If you use VisiData, I would love it if you reached out to me to discuss some of your common workflows and needs. This helps me better prioritize which functionality to add. Send me a [screencast](http://asciinema.org), or some sample data along with your desired output.  There is probably a way to tweak VisiData to get the job done even more to your liking.  Feature requests should be made on any of the communication channels listed [here](https://github.com/saulpw/visidata/blob/develop/CONTRIBUTING.md#community). 
+If you use VisiData, I would love it if you reached out to me to discuss some of your common workflows and needs. This helps me better prioritize which functionality to add. Send me a [screencast](http://asciinema.org), or some sample data along with your desired output.  There is probably a way to tweak VisiData to get the job done even more to your liking.  Feature requests should be made on any of the communication channels listed [here](https://github.com/saulpw/visidata/blob/stable/CONTRIBUTING.md#community). 
 
 ### Filing issues or support
 
-Create a GitHub issue if anything doesn't appear to be working right. If you get an unexpected error, please include the full stack trace that you get with `^E`. Additionally, it would help me more quickly diagnose the problem if you could attach the saved Commandlog (`Ctrl-D`), which will show the steps that lead to the issue. If you are struggling with learning how to use the tool, or are unsure how something works, please also file an issue or write a comment in any of our [community spaces](https://github.com/saulpw/visidata/blob/develop/CONTRIBUTING.md#community). In addition to wanting to help users get the most out of the tool, this helps us gauge the holes in our documentation.
+Create a GitHub issue if anything doesn't appear to be working right. If you get an unexpected error, please include the full stack trace that you get with `Ctrl-E`. Additionally, it would help me more quickly diagnose the problem if you could attach the saved Commandlog (`Ctrl-D`), which will show the steps that lead to the issue. If you are struggling with learning how to use the tool, or are unsure how something works, please also file an issue or write a comment in any of our [community spaces](https://github.com/saulpw/visidata/blob/stable/CONTRIBUTING.md#community). In addition to wanting to help users get the most out of the tool, this helps us gauge the holes in our documentation.
 
 ### Contributing tests
 
@@ -33,7 +33,7 @@ To run a test manually:
 
 ```
        $ bin/vd --play tests/foo.vd --replay-wait 1
-    or $ bin/vd -p tests/foo.vd -d 1
+    or $ bin/vd -p tests/foo.vd -w 1
 ```
 
 To build a `.vd` file:
@@ -42,15 +42,17 @@ To build a `.vd` file:
 2. Press `Shift-D` to view the commandlog.
 3. Edit the commandlog to minimize the number of commands.  Cells may be parameterized like `{foo}`, to be specified on the commandline:
 
-    $ vd cmdlog.vd --foo=value  
+```
+    $ vd cmdlog.vd --foo=value
+```
 
-4. Press `^S` to save the commandlog to a `.vd` file.
+4. Press `Ctrl-S` to save the commandlog to a `.vd` file.
 
-As a shortcut, `^D` will save the current commandlog, by default to the next non-existing 'cmdlog-#.vd'
+As a shortcut, `Ctrl-D` will save the current commandlog, by default to the next non-existing 'cmdlog-#.vd'
 
 ### Contributing to documentation
 
-If you notice a `globalCommand()` or `Command()` which does not have an entry in the VisiData manpage, please file an issue. In addition, if something is not clear (and in fact, is confusing) let us know so that we can better improve on the documentation.
+If you notice a `globalCommand()` or `Command()` which does not have an entry in the vd manpage, please file an issue. In addition, if something is not clear (and in fact, is confusing) let us know so that we can better improve on the documentation.
 
 If you would like to contribute by building an asciicast, the process is shown at [visidata.org/test/meta](http://visidata.org/test/meta).
 

README.md

@@ -1,4 +1,4 @@
-# VisiData v0.98.1 [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable)
+# VisiData v0.99 [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable)
 
 A terminal interface for exploring and arranging tabular data.
 
@@ -7,12 +7,21 @@ A terminal interface for exploring and arranging tabular data.
 - Linux or OS/X
 - Python 3.4+
 - python-dateutil
-- other [modules may be required](https://github.com/saulpw/visidata/blob/stable/requirements.txt) for opening particular data sources
-    - for a breakdown, see [supported sources](http://visidata.org/man/) in the VisiData manpage
+- other modules may be required for opening particular data sources
+    - see [requirements.txt](https://github.com/saulpw/visidata/blob/stable/requirements.txt) or the [supported sources](http://visidata.org/man/#loaders) in the vd manpage
 
-## Install
+## Installation
 
-To install VisiData, with loaders for formats supported by the Python standard library (which includes csv, tsv, fixed-width text, json, sqlite and graphs):
+There are three options for installing visidata:
+- [pip3](https://github.com/saulpw/visidata/tree/stable#install-via-pip3) for users who wish to import visidata into their own code or wish to integrate it into a python virtual environment
+- [Homebrew](https://github.com/saulpw/visidata/tree/stable#install-via-brew) on MacOS/X for reliable installation of application components (such as the manpage)
+- [apt](https://github.com/saulpw/visidata/tree/stable#install-via-apt) on Linux distributions
+
+### Install via pip3
+
+Best installation method for users who wish to take advantage of VisiData in their own code, or integrate it into a Python3 virtual environment.
+
+To install VisiData, with loaders for the most common data file formats (including csv, tsv, fixed-width text, json, sqlite, http, html and xls):
 
 ```
 $ pip3 install visidata
@@ -24,14 +33,53 @@ To install VisiData, plus external dependencies for all available loaders:
 pip3 install "visidata[full]"
 ```
 
+### Install via brew
+
+Ideal for MacOS users who primarily want to engage with VisiData as an application. This is currently the most reliable way to install VisiData's manpage on MacOS.
+
+```
+brew install saulpw/vd/visidata
+```
+
+Further instructions available [here](https://github.com/saulpw/homebrew-vd).
+
+### Install via apt
+
+Packaged for Linux users who do not wish to wrangle with PyPi or python3-pip.
+
+Currently, VisiData is undergoing review for integration into the main Debian repository. Until then it is available in our [Debian repo](https://github.com/saulpw/deb-vd).
+
+Grab our public key
+
+```
+wget http://visidata.org/devotees.gpg.key
+apt-key add devotees.gpg.key
+```
+
+Add our repository to apt's search list
+
+```
+sudo apt-get install apt-transport-https
+sudo vim /etc/apt/sources.list
+    deb[arch=amd64] https://raw.githubusercontent.com/saulpw/deb-vd/master sid main
+sudo apt-get update
+```
+You can then install VisiData by typing:
+
+```
+sudo apt-get install visidata
+```
+
+Further instructions available [here](https://github.com/saulpw/deb-vd).
+
 ## Run
 
 ```
 $ vd [<options>] <input> ...
 $ <command> | vd [<options>]
 ```
 
-VisiData supports tsv, csv, xlsx, hdf5, sqlite, and more.
+VisiData supports tsv, csv, xlsx, hdf5, sqlite, json and more.
 Use `-f <filetype>` to force a particular filetype.
 
 ## Documentation
@@ -50,21 +98,23 @@ For more detailed information about how you can contribute as a developer, influ
 
 ## vdtui
 
-The core `vdtui.py` can be used to quickly create efficient terminal workflows.
+The core `vdtui.py` can be used to quickly create efficient terminal workflows. These have been prototyped as proof of this concept:
 
 - [vgit](https://github.com/saulpw/vgit): a git interface
+- [vsh](http://github.com/saulpw/vsh): a collection of utilities like `vping` and `vtop`.
 - [vdgalcon](https://github.com/saulpw/vdgalcon): a port of the classic game [Galactic Conquest](https://www.galcon.com)
 
-Other workflows should also be created as separate apps using vdtui.  These apps can be very small; for example, see the included [viewtsv](bin/viewtsv).
+Other workflows should also be created as separate apps using vdtui.  These apps can be very small and provide a lot of functionality; for example, see the included [viewtsv](bin/viewtsv).
 
 
 ## License
 
 The innermost core file, `vdtui.py`, is a single-file stand-alone library that provides a solid framework for building text user interface apps. It is distributed under the MIT free software license, and freely available for inclusion in other projects.
 
-Other VisiData components, including the main `vd` application, addons, loaders, and other code in this repository, are available for reuse under GPLv3.
+Other VisiData components, including the main `vd` application, addons, loaders, and other code in this repository, are available for use and distribution under GPLv3.
 
 ## Credits
 
-VisiData was created by Saul Pwanson `<visidata@saul.pw>`.
-Thanks to @anjakefala for test and release support, to @databranner for documentation, and to those wonderful users who contribute feedback in any form, for helping to make VisiData the awesome tool that it is.
+VisiData was created and developed by Saul Pwanson `<vd@saul.pw>`.
+
+Thanks to all the [contributors](CONTRIBUTING.md#contributors), and to those wonderful users who provide feedback, for making VisiData the awesome tool that it is.

RELEASE.md

@@ -0,0 +1,126 @@
+# Release process for the next `stable` version
+
+1. Merge `stable` to `develop` (if necessary)
+
+2. Ensure `develop` automated tests run correctly with ./test.sh
+
+3. Verify that documentation/docstrings are up-to-date on features and functionality
+
+4. Verify that setup.py is up-to-date with requirements.
+
+5. Set version number to next most reasonable number (v#.#.#)
+
+   a. add to front of CHANGELOG, along with the release date and bullet points of major changes
+
+   b. update version number on README and front page of website
+
+   c. bump version in `__version__` in source code (bin/vd, visidata/vdtui.py) and setup.py
+
+6. Run ./mkwww.sh to build the manpage and updated website
+
+7. Push `develop` to testpypi
+
+    a. set up a ~/.pypirc
+
+    ```
+    [distutils]
+    index-servers=
+        pypi
+        testpypi
+    [pypi]
+    repository:https://upload.pypi.org/legacy/
+    username:
+    password:
+
+    [testpypi]
+    repository: https://test.pypi.org/legacy
+    username:
+    password:
+    ```
+
+    b. push to testpypi
+
+    ```
+    python3 setup.py sdist
+    twine upload dist/* -r testpypi
+    ```
+
+8. Test install from testpypi
+
+   a. on virgin instance
+
+   b. on mac and linux
+
+   c. See if windows works
+
+   d. from git clone
+
+   ```
+   pip3 install --extra-index-url https://test.pypi.org/project visidata
+   ```
+
+9. Merge `develop` to stable
+
+10. Merge `stable` back into other branches
+
+    a. if the branch works with minimal conflicts, keep the branch
+
+    b. otherwise, clean out the branch
+
+
+11. Push stable to pypi
+
+```
+twine upload dist/*
+```
+
+12. Test install/upgrade from pypi
+
+  a. Build and deploy the website
+
+   b. Ask someone else to test install
+
+13. Create a tag `v#.#.#` for that commit
+```
+git tag v#.#.#
+git push --tags
+```
+
+14. Push code to stable
+
+15. Write up the release notes and post at visidata.org/release/#.#
+
+16. Comb through issues and close the ones that have been solved, referencing the version number
+
+17. Post release notes on r/visidata and tinyletter and have some ice cream
+
+
+# Homebrew
+
+1. Update the link in url to the new visidata tar.gz file.
+2. Download the tarfile and obtain its new sha256
+```
+shasum -a 256
+```
+3. Check each dependency and see if it has been updated. If so, update the url and sha256 for the newest version.
+4. Install visidata using `pip3 install visidata --upgrade` and note down all of the new dependencies. 
+5. Obtain their urls and sha256 and add them to the formula
+6. Change their urls from `pypi.python.org` to `files.pythonhosted.org`.
+7. Test the formula with `brew install --build-from-source visidata`. Fix as needed.
+8. Audit the formula with `brew audit --new-formula visidata`
+9. Add and commit the formula.
+
+## Debian
+1. Obtain the visidata tar.gz file from pypi
+2. tar -xzmf visidata.tar.gz
+3. cp visidata.ver.tar.gz visidata_ver.orig.tar.gz
+4. cd visidata/
+5. Place there the contents of the debian directory from github.com/saulpw/visidata
+6. Update changelog
+```
+dch -v version-revision
+```
+7. Run debuild. Fix errors as they come up
+8. If a package fails to import a module, it must be added to the build dependencies as python3-modules
+9. Enter saulpw/deb-vd.
+10. Run the command reprepro includeb sid new-vd.deb