Can make real docs now (API and overview).

CHANGES.txt

@@ -1,3 +1,8 @@
+v1.1.0, 10/09/18 -- fixes to_dataframes and to_gdx memory leaks, respects 
+                    GAMS_DIR environment variable, additional bug and warning 
+                    fixes. GdxSymbol.dataframe can now be set just using the 
+                    GdxSymbol.dims columns; in that case the value columns are 
+                    filled in with defaults
 v1.0.4, 03/02/18 -- make sure set 'Value' column fix works even when dims are all named '*'
 v1.0.3, 02/15/18 -- fix up set 'Value' column when write to GDX
 v1.0.2, 01/19/18 -- instructions and additional filepaths for running pytest on 

README.md

@@ -1,112 +1,10 @@
 # gdx-pandas
 [![PyPI](https://img.shields.io/pypi/v/gdxpds.svg)](https://pypi.python.org/pypi/gdxpds/)
+[![Documentation](https://img.shields.io/badge/docs-ready-blue.svg)](http://nrel.github.io/gdx-pandas)
 
-Python package to translate between gdx (GAMS data) and pandas. 
+gdx-pandas is a python package to translate between gdx (GAMS data) and pandas. 
 
-[Install](#install) | [Use](#use) | [Uninstall](#uninstall)
-
-## Use
-
-There are two main ways to use gdxpds. The first use case is the one that was 
-initially supported: direct conversion between GDX files on disk and pandas 
-DataFrames or a csv version thereof. The Version 1.0.0 rewrite intoduces a 
-second style of use, that is, interfacing with GDX files and symbols via the 
-`gdxpds.gdx.GdxFile` and `gdxpds.gdx.GdxSymbol` classes.
-
-[Direct Conversion](#direct-conversion) | [Backend Classes](#backend-classes)
-
-### Direct Conversion
-
-The two primary points of reference for the direct conversion utilities are GDX 
-files on disk and python dicts of {symbol_name: pandas.DataFrame}, where 
-each pandas.DataFrame contains data for a single set, parameter, equation, or 
-variable. For sets and parameters, the last column of the DataFrame is assumed to
-contain the value of the element, which for sets should be `True`, and for 
-parameters should be a `float` (or one of the `gdxpds.gdx.NUMPY_SPECIAL_VALUES`). 
-Equations and variables have additional 'value' columns, in particular a level, 
-a marginal value, a lower bound, an upper bound, and a scale, as enumerated in 
-`gdxpds.gdx.GamsValueType`. These values are all assumed to be found in the last 
-five columns of the DataFrame, also see `gdxpds.gdx.GAMS_VALUE_COLS_MAP`.
-
-The basic interface to convert from GDX to DataFrames is:
-
-```python
-import gdxpds
-
-gdx_file = 'C:\path_to_my_gdx\data.gdx'
-dataframes = gdxpds.to_dataframes(gdx_file)
-for symbol_name, df in dataframes.items():
-    print("Doing work with {}.".format(symbol_name))
-```
-
-And vice-versa:
-
-```python
-import gdxpds
-
-# assume we have a DataFrame df with last column 'value'
-data_ready_for_GAMS = { 'symbol_name': df }
-
-gdx_file = 'C:\path_to_my_output_gdx\data_to_send_to_gams.gdx'
-gdx = gdxpds.to_gdx(data_ready_for_GAMS, gdx_file)
-```
-
-Note that providing a gdx_file is optional, and the returned gdx is an object of 
-type `gdxpds.gdx.GdxFile`.
-
-Additional functions include:
-
-- `gdxpds.list_symbols`
-- `gdxpds.to_dataframe` (If the call to this method includes 
-   old_interface=False, then the return value will be a plain DataFrame, not a 
-   {'symbol_name': df} dict.)
-
-The package also includes command line utilities for converting between GDX and 
-CSV, see
-
-```bash
-python C:\your_python_path\Scripts\gdx_to_csv.py --help
-python C:\your_python_path\Scripts\csv_to_gdx.py --help
-```
-
-### Backend Classes
-
-The basic functionalities described above can also be achieved with direct use
-of the backend classes now available in `gdxpds.gdx`. To duplicate the GDX read 
-functionality shown above one would write:
-
-```python
-import gdxpds
-
-gdx_file = 'C:\path_to_my_gdx\data.gdx'
-with gdxpds.gdx.GdxFile(lazy_load=False) as f:
-    f.read(gdx_file)
-    for symbol in f:
-        symbol_name = symbol.name
-        df = symbol.dataframe
-        print("Doing work with {}:\n{}".format(symbol_name,df.head()))
-```
-
-The backend especially gives more control over creating new data in GDX format. 
-For example:
-
-```python
-import gdxpds
-
-out_file = 'my_new_gdx_data.gdx'
-with gdxpds.gdx.GdxFile() as gdx:
-    # Create a new set with one dimension
-    gdx.append(gdxpds.gdx.GdxSymbol('my_set',gdxpds.gdx.GamsDataType.Set,dims=['u']))
-    data = pds.DataFrame([['u' + str(i)] for i in range(1,11)])
-    data['Value'] = True
-    gdx[-1].dataframe = data
-    # Create a new parameter with one dimension
-    gdx.append(gdxpds.gdx.GdxSymbol('my_parameter',gdxpds.gdx.GamsDataType.Parameter,dims=['u']))
-    data = pds.DataFrame([['u' + str(i), i*100] for i in range(1,11)],
-                         columns=(gdx[-1].dims + gdx[-1].value_col_names))
-    gdx[-1].dataframe = data
-    gdx.write(out_file)
-```
+[Install](#install) | [Uninstall](#uninstall)
 
 ## Install
 
@@ -115,8 +13,6 @@ with gdxpds.gdx.GdxFile() as gdx:
 - Python 2.6 or higher 2.X; Python 3.4 or higher 3.X
 - pandas (In general you will want the SciPy stack. Anaconda comes with it, or see [my notes for Windows](http://elainethale.wordpress.com/programming-notes/python-environment-set-up/).)
 - For Python versions < 3.4, enum34. Also **uninstall the enum package** if it is installed.
-- psutil (optional--for monitoring memory use)
-- pytest (optional--for running tests)
 - GAMS Python bindings
     - See GAMS/win64/XX.X/apifiles/readme.txt on Windows, 
       GAMS/gamsXX.X_osx_x64_64_sfx/apifiles/readme.txt on Mac, or 
@@ -151,7 +47,7 @@ pip install gdxpds
 or
 
 ```bash
-pip install git+https://github.com/NREL/gdx-pandas.git@v1.0.4
+pip install git+https://github.com/NREL/gdx-pandas.git@v1.1.0
 ```
 
 or

dev/README.md

@@ -1,15 +1,96 @@
 # Developer How-To
 
+To get all of the development dependencies for Python:
+
+```
+pip install -r layerstack/dev/requirements.txt
+```
+
+Also, you will need to install
+
+- [pandoc](https://pandoc.org/installing.html)
+
 ## Create a new release
 
-- Make the release on github
-- Install from github and make sure tests pass
-- Create package and push to pypi:
+1. Update version number, CHANGES.txt, setup.py, LICENSE and header as needed
+2. Run tests locally and fix any issues
+3. Install from github and make sure tests pass 
+4. Uninstall the draft package
+5. Publish documentation
+6. Create release on github
+7. Release tagged version on pypi
+
+## Publish documentation
+
+The documentation is built with [Sphinx](http://sphinx-doc.org/index.html). There are several steps to creating and publishing the documentation:
+
+1. Convert .md input files to .rst
+2. Refresh API documentation
+3. Build the HTML docs
+4. Push to GitHub
+
+### Markdown to reStructuredText
+
+Markdown files are registered in `docs/source/md_files.txt`. Paths in that file should be relative to the docs folder and should exclude the file extension. For every file listed there, the `dev/md_to_rst.py` utility will expect to find a markdown (`.md`) file, and will look for an optional `.postfix` file, which is expected to contain `.rst` code to be appended to the `.rst` file created by converting the input `.md` file. Thus, running `dev/md_to_rst.py` on the `doc/source/md_files.txt` file will create revised `.rst` files, one for each entry listed in the registry. In summary:
+
+```
+cd doc/source
+python ../../dev/md_to_rst.py md_files.txt
+```
+
+### Refresh API Documentation
+
+- Make sure layerstack is in your PYTHONPATH
+- Delete the contents of `source/api`.
+- Run `sphinx-apidoc -o source/api ..` from the `doc` folder.
+- Compare `source/api/modules.rst` to `source/api.rst`. Delete `setup.rst` and references to it.
+- 'git push' changes to the documentation source code as needed.
+- Make the documentation per below
+
+### Building HTML Docs
+
+Run `make html` for Mac and Linux; `make.bat html` for Windows.
+
+### Pushing to GitHub Pages
+
+#### Mac/Linux
+
+```
+make github
+```
+
+#### Windows
+
+```
+make.bat html
+```
+
+Then run the github-related commands by hand:
+
+```
+git branch -D gh-pages
+git push origin --delete gh-pages
+ghp-import -n -b gh-pages -m "Update documentation" ./build/html
+git checkout gh-pages
+git push origin gh-pages
+git checkout master # or whatever branch you were on
+```
+
+## Release on pypi
+
+1. [using testpyi](https://packaging.python.org/guides/using-testpypi/) has good instructions for setting up your user account on TestPyPI and PyPI, and configuring twine to know how to access both repositories.
+2. Test the package
 
     ```
     python setup.py sdist
     twine upload --repository testpypi dist/*
     # look at https://test.pypi.org/project/gdxpds/
+    pip install --index-url https://test.pypi.org/simple/gdxpds
+    # check it out ... fix things ...
+    ```
+
+3. Upload to pypi
+
+    ```
     twine upload --repository pypi dist/*
     ```
-    

dev/md_to_rst.py

@@ -0,0 +1,66 @@
+import argparse
+import logging
+import os
+from subprocess import call, list2cmdline
+
+from layerstack import start_console_log
+
+logger = logging.getLogger(__name__)
+
+def convert_files(file_registry):
+    # registry is expected to contain paths relative to its location
+    base_path = os.path.dirname(file_registry)
+
+    if not os.path.exists(file_registry):
+        raise ValueError("File registry {} not found".format(file_registry))
+
+    # loop through registry of md files
+    with open(file_registry,'r') as registry:
+        for line in registry:
+            if line:
+                # non-empty
+                p = os.path.join(base_path,line.strip())
+                p_md = p + '.md'
+                if not os.path.exists(p_md):
+                    raise ValueError("There is no {} file.".format(p_md))
+                # run pandoc
+                p_rst = p + '.rst'
+                try:
+                    cmd_and_args = ['pandoc',p_md,'-o',p_rst]
+                    call(cmd_and_args)
+                except Exception as e:
+                    try:
+                        call(['pandoc'])
+                    except:
+                        logger.error("Call to pandoc fails")
+                        raise e
+                    if not os.path.exists(p_md):
+                        logger.error("Input file {} does not exist".format(p_md))
+                        raise e
+                    logger.error("Call '{}' failed".format(list2cmdline(cmd_and_args)))
+                    raise e
+                # append .postfix
+                p_postfix = p + '.postfix'
+                if os.path.exists(p_postfix):
+                    with open(p_rst,'a') as rst:
+                        rst.write("\n")
+                        with open(p_postfix,'r') as postfix:
+                            rst.write(postfix.read())
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="""Utility to convert Markdown 
+        (.md) files to reStructuredText (.rst)""")
+    parser.add_argument('file_registry',help="""Text file that lists the 
+        markdown files to convert. Each line is the file path and name for an 
+        .md file, where the path is relative to the location of file_registry, 
+        and the .md extension is omitted.""")
+    parser.add_argument("-d","--debug",action='store_true',default=False,
+        help="Option to output debug information.")
+
+    args = parser.parse_args()
+
+    # start logging
+    start_console_log(log_level=logging.DEBUG if args.debug else logging.INFO)
+
+    # perform conversion
+    convert_files(args.file_registry)

dev/requirements.txt

@@ -1,2 +1,7 @@
-sphinx
+ghp-import
+numpydoc
 pandoc
+pytest
+sphinx
+sphinx_rtd_theme
+twine

doc/source/api.rst

@@ -0,0 +1,7 @@
+API
+###
+
+.. toctree::
+   :maxdepth: 4
+
+   api/gdxpds

doc/source/api/gdxpds.rst

@@ -0,0 +1,53 @@
+gdxpds package
+==============
+
+Subpackages
+-----------
+
+.. toctree::
+
+    gdxpds.test
+
+Submodules
+----------
+
+gdxpds.gdx module
+-----------------
+
+.. automodule:: gdxpds.gdx
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+gdxpds.read\_gdx module
+-----------------------
+
+.. automodule:: gdxpds.read_gdx
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+gdxpds.tools module
+-------------------
+
+.. automodule:: gdxpds.tools
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+gdxpds.write\_gdx module
+------------------------
+
+.. automodule:: gdxpds.write_gdx
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: gdxpds
+    :members:
+    :undoc-members:
+    :show-inheritance: