diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 000000000..d74c537f6 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,52 @@ +--- +name: Bug report +about: Create a report to help us fix mistakes in the user guide +title: '' +labels: '' +assignees: '' + +--- + +**Describe the bug** +A clear and concise description of what the bug is. + +**To Reproduce** +Steps to reproduce the behavior: +1. Go to '...' +2. Click on '....' +3. Scroll down to '....' +4. See error + +**Expected behavior** +A clear and concise description of what you expected from the user guide. Please include a link to the any sections that you reference. + +**Actual behavior** +What is there, or not there, instead. Add as much detail as you can. Include (copy and paste) error messages, stack traces, and output of any code that is related. + +If applicable, show us how to reproduce the failure. If you can, use trajectory files from the test data. + +``` python +import MDAnalysis as mda +from MDAnalysis.tests.datafiles import PSF, DCD, GRO, PDB, TPR, XTC, TRR, PRMncdf, NCDF + +u = mda.Universe(PSF, DCD) + +.... + +``` + +**Desktop (please complete the following information):** + - OS: [e.g. iOS] + - Browser [e.g. chrome, safari] + - Version [e.g. 22] +- Which version of MDAnalysis? (run `python -c "import MDAnalysis as mda; print(mda.__version__)"`) +- Which version of Python (`python -V`)? + +**Smartphone (please complete the following information):** + - Device: [e.g. iPhone6] + - OS: [e.g. iOS8.1] + - Browser [e.g. stock browser, safari] + - Version [e.g. 22] + +**Additional context** +Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 000000000..6900eaf99 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,20 @@ +--- +name: Feature request +about: Suggest an idea for this project +title: '' +labels: '' +assignees: '' + +--- + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] + +**Describe the solution you'd like** +A clear and concise description of what you want to see in the user guide. e.g. a new page of information? A short Jupyter notebook example? An extended tutorial? + +**Describe alternatives you've considered** +A clear and concise description of any alternative solutions or features you've considered. + +**Additional context** +Add any other context or screenshots about the feature request here. diff --git a/.github/ISSUE_TEMPLATE/questions.md b/.github/ISSUE_TEMPLATE/questions.md new file mode 100644 index 000000000..a2814f71a --- /dev/null +++ b/.github/ISSUE_TEMPLATE/questions.md @@ -0,0 +1,27 @@ +--- +name: Questions +about: If you want to ask a question please use the mailing list! +title: '' +labels: '' +assignees: '' + +--- + +If you have a **QUESTION** such as + +- how to use MDAnalysis in general +- how to accomplish something specific, +- what output means +- ... or similar questions related to USING MDAnalysis + +then please *ask this question on the user mailing list* + + https://groups.google.com/forum/#!forum/mdnalysis-discussion + +The issue tracker is meant for DEFECTS ("BUGS"), new FEATURES, and decisions on the direction of the user guide. In order to keep the volume of work on the issue tracker manageable for our volunteer developers, we will **immediately close issues that are better answered on the user mailing list.** + +We really appreciate you as a user getting in touch with us --- no matter what you want to discuss. But we need your help keeping the amount of work manageable so please use the mailing list for questions and the issue tracker for code-related issues. + +Thank you! + +The MDAnalysis Development Team diff --git a/doc/source/conf.py b/doc/source/conf.py index 8c3327124..6b664c65f 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -72,9 +72,11 @@ # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. + exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', '.ipynb_checkpoints', '**/.ipynb_checkpoints', 'scripts', '.*.ipynb'] + # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for @@ -147,6 +149,7 @@ intersphinx_mapping = {'https://docs.python.org/': None, 'https://docs.scipy.org/doc/numpy/': None, 'https://www.mdanalysis.org/docs/': None, + 'https://docs.pytest.org/en/latest/': None, } # nbsphinx diff --git a/doc/source/contributing.rst b/doc/source/contributing.rst index 195f882db..d47e0f821 100644 --- a/doc/source/contributing.rst +++ b/doc/source/contributing.rst @@ -10,8 +10,11 @@ The development team very much welcomes contributions from its users. Contributions can take many forms, such as: * **bug reports** or **enhancement requests** filed through the `Issue Tracker`_ - * **source code** for fixing bugs or new functionality (e.g. new analysis tasks) - * **documentation** (either in the code or on the user guide) + * **bug fixes** + * **improvements** to the code (speed, clarity, modernised code) + * **new features** in the code + * improvements and additions to **documentation** (including typo fixes) + * improvements to the **build systems** * **questions** or **discussions** on the `mdnalysis-discussion`_ mailing list The MDAnalysis community subscribes to a `Code of Conduct`_ that all community @@ -30,32 +33,8 @@ All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome. If you are brand new to MDAnalysis or open-source development, we recommend going -through this page. If you are new to Git and version control, have a look at -:ref:`version-control`. Get started working with the code of -:ref:`the main MDAnalysis codebase ` or -:ref:`the documentation `. - -A brief overview of the workflow for contributing to MDAnalysis is laid out below, but it is recommended to read the whole guide for the particular section you are interested in. - -The development workflow for code or inline code documentation looks like this: - - #. :ref:`Fork the MDAnalysis repository ` from the mdanalysis account into your own account - #. :ref:`Set up an isolated virtual environment ` for code development - #. :ref:`Build development versions ` of MDAnalysis and MDAnalysisTests on your computer into the virtual environment - #. :ref:`Create a new branch off the develop branch ` - #. :ref:`Add your new feature or bug fix ` or :ref:`add your new documentation ` - #. :ref:`Add and run tests ` (if adding to the code) - #. :ref:`Build and view the documentation ` (if adding to the docs) - #. :ref:`Commit and push your changes, and open a pull request. ` - -The workflow for working with user guide looks like this: - - #. :ref:`Fork the MDAnalysis repository ` from the mdanalysis account into your own account - #. :ref:`Set up an isolated virtual environment ` for your documentation - #. :ref:`Create a new branch off the master branch ` - #. Add your new documentation. - #. :ref:`Build and view the documentation `. - #. :ref:`Commit and push your changes, and open a pull request `. +through the guides for :ref:`contributing to the main codebase ` or the :ref:`user guide `. If you are new to Git and version control, have a look at +:ref:`version-control`. .. _version-control: @@ -88,821 +67,7 @@ Getting started with Git setting up your SSH key, and configuring git. All these steps need to be completed before you can work seamlessly between your local repository and GitHub. -.. _working-with-mdanalysis-code: - -Working with the code -===================== - -.. _forking-code-repo: - -------- -Forking -------- - -You will need your own fork to work on the code. Go to the `MDAnalysis project page `_ and hit the :guilabel:`Fork` button. You will -want to clone your fork to your machine: - -.. code-block:: bash - - git clone https://github.com/your-user-name/mdanalysis.git - cd mdanalysis - git remote add upstream https://github.com/MDAnalysis/mdanalysis - -This creates the directory `mdanalysis` and connects your repository to -the upstream (main project) MDAnalysis repository. - -.. _create-virtual-environment: - ----------------------------------- -Creating a development environment ----------------------------------- - -To change code and test changes, you'll need to build both **MDAnalysis** and **MDAnalysisTests** -from source. This requires a Python environment. We highly recommend that you use -virtual environments. This allows you to have multiple experimental development versions -of MDAnalysis that do not interfere with each other, or your own stable version. -Since MDAnalysis is split into the actual package and a test suite, you need to install -both modules in development mode. - -You can do this either with :ref:`conda ` or :ref:`pip `. - -.. _dev-with-conda: - -With conda ----------- - -Install either `Anaconda `_ -or `miniconda `_. -Make sure your conda is up to date: - - .. code-block:: bash - - conda update conda - -Create a new environment with ``conda create``. This will allow you to change code in -an isolated environment without touching your base Python installation, and without -touching existing environments that may have stable versions of MDAnalysis. : - - .. code-block:: bash - - conda create --name mdanalysis-dev - -Activate the environment to build MDAnalysis into it: - - .. code-block:: bash - - conda activate mdanalysis-dev - -To view your environments: - - .. code-block:: bash - - conda info -e - -To list the packages installed in your current environment: - - .. code-block:: bash - - conda list - -To return to your root environment: - - .. code-block:: bash - - conda deactivate - -See the full conda docs `here `__. - -.. _dev-with-pip: - -With pip and virtualenv ------------------------ - -Like conda, virtual environments managed with `virtualenv `_ allow you to use different versions of Python and Python packages for your different project. Unlike conda, virtualenv is not a general-purpose package manager. Instead, it leverages what is available on your system, and lets you install Python packages using pip. - -To use virtual environments you have to install the virtualenv package first. This can be done with either pip or the package manager of your system: - - .. code-block:: bash - - pip install virtualenv - # or on ubuntu - sudo apt install virtualenv - # or on fedora - sudo dnf install python-virtualenv - -Virtual environments can be created for each project directory. - - .. code-block:: bash - - cd my-project/ - virtualenv my-project-env - -This will create a new folder ``my-project-env``. This folder contains the virtual environment and all packages you have installed in it. To activate it in the current terminal run: - - .. code-block:: bash - - source myproject-env/bin/activate - -Now you can install packages via pip without affecting your global environment. The packages that you install when the environment is activated will be available in terminal sessions that have the environment activated. You can deactivate the virtual environment by running: - - .. code-block:: bash - - deactivate - -The `virtualenvwrapper package `_ makes virtual environments easier to use. It provides some very useful features: - - - it organises the virtual environment into a single user-defined directory, so they are not scattered throughout the file system; - - it defines commands for the easy creation, deletion, and copying of virtual environments; - - it defines a command to activate a virtual environment using its name; - - all commands defined by ``virtualenvwrapper`` have tab-completion for virtual environment names. - -You first need to install ``virtualenvwrapper`` *outside* of a virtual environment: - - .. code-block:: bash - - pip install virtualenvwrapper - # or on ubuntu - sudo apt install virtualenvwrapper - # or on fedora - sudo dnf install python-virtualenvwrapper - -Then, you need to load it into your terminal session. Add the following lines in ``~/.bashrc``. They will be executed every time you open a new terminal session: - - .. code-block:: bash - - # Decide where to store the virtual environments - export WORKON_HOME=~/Envs - # Make sure the directory exists - mkdir -p ${WORKON_HOME} - # Load virtualenvwrapper - source /usr/local/bin/virtualenvwrapper.sh - -Open a new terminal or run ``source ~/.bashrc`` to update your session. You can now create a virtual environment with: - - .. code-block:: bash - - mkvirtualenv my-project - -Regardless of your current working directory, the environment is created in ``~/Envs/`` and it is now loaded in our terminal session. - -You can load your virtual environments by running ``workon my-project``, and exit them by running ``deactivate``. - -Virtual environments, especially with ``virtualenvwrapper``, can do much more. For example, you can create virtual environments with different python interpreters with the ``-p`` flag. The Hitchhiker's Guide to Python has a good `tutorial `_ that gives a more in-depth explanation of virtual environments. The `virtualenvwrapper documentation `_ is also a good resource to read. - -On a Mac --------- - -One more step is often required on macOS, because of the default number of files that a process can open simultaneously is quite low (256). To increase the number of files that can be accessed, run the following command: - - .. code-block:: bash - - ulimit -n 4096 - -This sets the number of files to 4096. However, this command only applies to your currently open terminal session. To keep this high limit, add the above line to your ``~/.profile``. - - - -.. _build-mdanalysis-develop: - -------------------- -Building MDAnalysis -------------------- - -Make sure that you have :ref:`cloned the repository ` -and activated your virtual environment. First we need to install dependencies: - - .. code-block:: bash - - # if using conda - conda install cython numpy - # if using pip - pip install cython numpy - -Ensure that you have a working C/C++ compiler (e.g. gcc or clang). You will also need Python 2.7.x or Python ≥ 3.4. We will now install MDAnalysis. - - .. code-block:: bash - - # go to the mdanalysis source directory - cd mdanalysis/ - - # Build and install the MDAnalysis package - cd package/ - python setup.py develop - - # Build and install the test suite - cd ../testsuite/ - python setup.py develop - -At this point you should be able to import MDAnalysis from your locally built version: - - .. code-block:: bash - - $ python # start an interpreter - >>> import MDAnalysis as mda - >>> mda.__version__ - '0.20.2-dev0' - -.. _branches-in-mdanalysis: - ----------------------- -Branches in MDAnalysis ----------------------- - -There are two important branches in MDAnalysis: - - - ``master``: for production-ready code - - ``develop``: for development code - -The ``master`` branch is only for stable, production-ready code. Development code should *never* be committed to this branch. Typically, code is only committed by the release manager, when a release is ready. - -The ``develop`` branch can be considered an "integration" branch for including your code into the next release. Only working, tested code should be committed to this branch. Code contributions ("features") should branch off ``develop`` rather than ``master``. - - -.. _create-code-branch: - -Creating a branch ------------------ - -The develop branch should only contain approved, tested code, so create a -feature branch for making your changes. For example, to create a branch called -``shiny-new-feature`` from ``develop``: - - .. code-block:: bash - - git checkout -b shiny-new-feature develop - -This changes your working directory to the ``shiny-new-feature`` branch. Keep any -changes in this branch specific to one bug or feature so it is clear -what the branch brings to MDAnalysis. You can have many branches with different names -and switch in between them using the ``git checkout my-branch-name`` command. - -There are several special branch names that you should not use for your feature branches: - - - ``master`` - - ``develop`` - - ``release-*`` - - -``release`` branches are used to :ref:`prepare a new production release ` and should be handled by the release manager only. - -.. _writing-new-code: - ----------------- -Writing new code ----------------- - -Code formatting in Python -------------------------- - -MDAnalysis is a project with a long history and many contributors; it hasn't used a consistent coding style. Since version 0.11.0, we are trying to update all the code to conform with `PEP8`_. Our strategy is to update the style every time we touch an old function and thus switch to `PEP8`_ continuously. - -**Important requirements (from PEP8):** - - keep line length to **79 characters or less**; break long lines sensibly - - indent with **spaces** and use **4 spaces per level** - - naming: - - - classes: `CapitalClasses` (i.e. capitalized nouns without spaces) - - methods and functions: `underscore_methods` (lower case, with underscores for spaces) - -We recommend that you use a Python Integrated Development Environment (IDE) (`PyCharm`_ and others) or external tools like `flake8`_ for code linting. For integration of external tools with emacs and vim, check out `elpy`_ (emacs) and `python-mode`_ (vim). - -To apply the code formatting in an automated way, you can also use code formatters. External tools include `autopep8`_ and `yapf`_. Most IDEs either have their own code formatter or will work with one of the above through plugins. - -Modules and dependencies ------------------------- - -MDAnalysis strives to keep dependencies small and lightweight. Code outside the :mod:`MDAnalysis.analysis` and :mod:`MDAnalysis.visualization` modules should only rely on the :ref:`core dependencies `, which are always installed. Analysis and visualization modules can use any :ref:`any package, but the package is treated as optional `. - -Imports in the code should follow the :ref:`general-rules-for-importing`. - -.. seealso:: - - See :ref:`module-imports` for more information. - - -Developing in Cython --------------------- - -The ``setup.py`` script first looks for the `.c` files included in the standard MDAnalysis distribution. These are not in the GitHub repository, so ``setup.py`` will use Cython to compile extensions. `.pyx` source files are used instead of `.c` files. From there, `.pyx` files are converted to `.c` files if they are newer than the already present `.c` files or if the ``--force`` flag is set (i.e. ``python setup.py build --force``). End users (or developers) should not trigger the `.pyx` to `.c` conversion, since `.c` files delivered with source packages are always up-to-date. However, developers who work on the `.pyx` files will automatically trigger the conversion since `.c` files will then be outdated. - -Place all source files for compiled shared object files into the same directory as the final shared object file. - -`.pyx` files and cython-generated `.c` files should be in the same directory as the `.so` files. External dependent C/C++/Fortran libraries should be in dedicated ``src/`` and ``include/`` folders. See the following tree as an example: - - :: - - MDAnalysis - |--lib - | |-- _distances.so - | |-- distances.pyx - | |-- distances.c - |-- coordinates - |-- _dcdmodule.so - |-- src - |-- dcd.c - |-- include - |-- dcd.h - - ------------------ -Testing your code ------------------ - -MDAnalysis takes testing seriously. All code added to MDAnalysis should have tests to ensure that it works as expected; we aim for 90% coverage. See :ref:`testing` for more on writing, running, and interpreting tests. - - ---------------------- -Documenting your code ---------------------- - -Changes to the code should be reflected in the ongoing ``CHANGELOG``. Add an entry here to document your fix, enhancement, or change. In addition, add your name to the author list. If you are addressing an issue, make sure to include the issue number. - - -.. _adding-code-to-mda: - ------------------------------- -Adding your code to MDAnalysis ------------------------------- - -Committing your code --------------------- - -When you are happy with a set of changes, it is time to commit. All changes in one revision should have a common theme. If you implemented two rather different things (say, one bug fix and one new feature), then split them into two commits with different messages. - -Once you’ve made changes to files in your local repository, you can see them by typing: - - .. code-block:: bash - - git status - -Tell git to track files by typing: - - .. code-block:: - - git add path/to/file-to-be-added.py - -Doing ``git status`` again should give something like: - - .. code-block:: - - # On branch shiny-new-feature - # - # modified: /relative/path/to/file-you-added.py - # - -Then commit with: - - .. code-block:: bash - - git commit -m - -This opens up a message editor. - -*Always* add a descriptive comment for your commit message (feel free to be verbose!): - - - use a short (<50 characters) subject line that summarizes the change - - leave a blank line - - optionally, add additional more verbose descriptions; paragraphs or bullet lists (with - or *) are good - - manually break lines at 80 characters - - manually indent bullet lists - -.. seealso:: - - See `Tim Pope's A Note About Git Commit Messages `_ for a rationale for these rules. - - -Pushing your code to GitHub ---------------------------- - -When you want your changes to appear publicly on your GitHub page, push your forked feature branch’s commits: - - .. code-block:: bash - - git push origin shiny-new-feature - -Here `origin` is the default name given to your remote repository on GitHub. You can see the remote repositories: - - .. code-block:: bash - - git remote -v - -If you added the upstream repository as described above you will see something like: - - .. code-block:: bash - - origin git@github.com:your-username/mdanalysis.git (fetch) - origin git@github.com:your-username/mdanalysis.git (push) - upstream git@github.com:MDAnalysis/mdanalysis.git (fetch) - upstream git@github.com:MDAnalysis/mdanalysis.git (push) - -Now your code is on GitHub, but it is not yet a part of the MDAnalysis project. For that to happen, a pull request needs to be submitted on GitHub. - -.. _rebase-code: - -Rebasing your code ------------------- - -Often the upstream MDAnalysis develop branch will be updated while you are working on your own code. -You will then need to update your own branch with the new code to avoid merge conflicts. -You need to first retrieve it and then `rebase `_ -your branch so that your changes apply to the new code: - - .. code-block:: bash - - git fetch upstream - git rebase upstream/develop - -This will replay your commits on top of the latest development code from MDAnalysis. If this -leads to merge conflicts, you must resolve these before submitting your pull -request. If you have uncommitted changes, you will need to ``git stash`` them -prior to updating. This will effectively store your changes and they can be -reapplied after updating with ``git stash apply``. - -Once rebased, push your changes: - - .. code-block:: bash - - git push -f origin shiny-new-feature - -and `create a pull request `_. - -.. _create-a-pull-request: - -Creating a pull request ------------------------ - -The typical approach to adding your code to MDAnalysis is to make a `pull request`_ on GitHub: - - #. Navigate to your repository on GitHub - #. Click on the :guilabel:`Pull Request` button - #. You can then click on :guilabel:`Commits` and :guilabel:`Files Changed` to make sure everything looks okay one last time - #. Write a description of your changes and follow the PR checklist - - - check that docs are updated - - check that tests run - - check that you've updated CHANGELOG - - reference the issue that you address, if any - - #. Click :guilabel:`Send Pull Request`. - -Your pull request is then sent to the repository maintainers. After this, the following happens: - - #. A :ref:`suite of tests are run on your code ` with the tools :ref:`travis`, :ref:`appveyor` and :ref:`codecov`. If they fail, please fix your pull request by pushing updates to it. - #. Developers will ask questions and comment in the pull request. You may be asked to make changes. - #. When everything looks good, a core developer will merge your code into the ``develop`` branch of MDAnalysis. Your code will be in the next release. - -If you need to make changes to your code, you can do so on your local repository as you did before. Committing and pushing the changes will update your pull request and restart the automated tests. - -.. _working-with-mdanalysis-docs: - -Working with the documentation -============================== - -MDAnalysis documentation is written in `reStructuredText `_ ("rst" or "reST") and built using `Sphinx`_. The -Sphinx Documentation has an excellent `introduction to reST -`__. - -MDAnalysis maintains two kinds of documentation: - - #. `This user guide `__: a map of how MDAnalysis works, combined with tutorial-like overviews of specific topics (such as the analyses) - - #. `The documentation generated from the code itself `__. Largely built from code docstrings, these are meant to provide a clear explanation of the usage of individual classes and functions. They often include technical or historical information such as in which version the function was added, or deprecation notices. - ---------------------------- -Working with the user guide ---------------------------- - -The user guide makes use of a number of Sphinx extensions to ensure that the code examples are always up-to-date. These include `nbsphinx `_ and the `ipython directive `__. - -The ``ipython`` directive lets you put code in the documentation which will be run -during the doc build. For example: - - :: - - .. ipython:: python - - x = 2 - x**3 - -will be rendered as: - - .. ipython:: - - In [1]: x = 2 - - In [2]: x**3 - Out[2]: 8 - -Many code examples in the docs are run during the -doc build. This approach means that code examples will always be up to date, -but it does make the doc building a bit more complex. - -.. _forking-user-guide: - -Forking and cloning the User Guide ----------------------------------- - -Go to the `MDAnalysis project page `_ and hit the :guilabel:`Fork` button. You will -want to clone your fork to your machine: - - .. code-block:: bash - - git clone https://github.com/your-user-name/UserGuide.git - cd UserGuide - git remote add upstream https://github.com/MDAnalysis/UserGuide - -This creates the directory `UserGuide` and connects your repository to -the upstream (main project) MDAnalysis repository. - - -.. _create-virtual-environment-user-guide: - -Creating a development environment ----------------------------------- - -:ref:`Create a new virtual environment ` for the user guide. Install the required dependencies, and activate the ``nglview`` extension. We use ``nglview`` for visualizing molecules in Jupyter notebook tutorials. - -If using conda: - - .. code-block:: bash - - cd UserGuide/ - conda env create python=3.6 -f environment.yml --quiet - conda activate mda-user-guide - jupyter-nbextension enable nglview --py --sys-prefix - - -If using pip: - - .. code-block:: bash - - cd UserGuide/ - pip install -r requirements.txt - jupyter-nbextension enable nglview --py --sys-prefix - -.. _build-user-guide: - -Building the user guide ------------------------ - -Navigate to the ``doc/`` directory and run ``make html``: - - .. code-block:: bash - - cd doc/ - make html - -The HTML output will be in ``doc/build/``, which you can open in your browser of choice. The homepage is ``doc/build/index.html``. - -If rebuilding the documentation becomes tedious after a while, install the :ref:`sphinx-autobuild ` extension. - -Saving state in Jupyter notebooks ---------------------------------- - -One of the neat things about ``nglview`` is the ability to interact with molecules via the viewer. This ability can be preserved for the HTML pages generated from Jupyer notebooks by ``nbsphinx``, if you save the notebook widget state after execution. - -.. _add-changes-user-guide: - -Adding changes to the UserGuide -------------------------------- - -As with the code, :ref:`commit and push ` your code to GitHub. Then :ref:`create a pull request `. The only test run for the User Guide is: that your file compile into HTML documents without errors. As usual, a developer will review your PR and merge the code into the User Guide when it looks good. - - ------------------------------------ -Working with the code documentation ------------------------------------ - -MDAnalysis has a lot of documentation in the Python doc strings. The docstrings follow the `Numpy Docstring Standard `__, which is used widely -in the Scientific Python community. They are nice to read as normal text and are converted by sphinx to normal ReST through `napoleon `__. - -This standard specifies the format of -the different sections of the docstring. See `this document -`_ -for a detailed explanation, or look at some of the existing functions to -extend it in a similar manner. - -Note that each page of the `online documentation `_ has a link to the *Source* of the page. You can look at it in order to find out how a particular page has been written in reST and copy the approach for your own documentation. - -.. _building-code-documentation: - -Building the documentation --------------------------- - -The online documentation is generated from the pages in ``mdanalysis/package/doc/sphinx/source/documentation_pages``. The documentation for the current release are hosted at www.mdanalysis.org/docs, while the development version is at www.mdanalysis.org/mdanalysis/. - -In order to build the documentation, you must first :ref:`clone the main MDAnalysis repo `. :ref:`Set up a virtual environment ` in the same way as you would for the code (you can use the same environment as you do for the code). You will need to install several packages for the docs. - - .. code-block:: bash - - pip install sphinx sphinx-sitemap sphinx_rtd_theme - -In addition, build the development version of MDAnalysis (if you haven't done this already): - - .. code-block:: bash - - python setup.py develop - -Then, generate the docs with: - - .. code-block:: bash - - python setup.py build_sphinx -E - -This generates and updates the files in ``doc/html``. If the above command fails with an ``ImportError``, run - - .. code-block:: bash - - python setup.py build_ext --inplace - -and retry. - -You will then be able to open the home page, ``doc/html/index.html``, and look through the docs. In particular, have a look at any pages that you tinkered with. It is typical to go through multiple cycles of fix, rebuild the docs, check and fix again. - -If rebuilding the documentation becomes tedious after a while, install the :ref:`sphinx-autobuild ` extension. - -Where to write docstrings -------------------------- - -When writing Python code, you should always add a docstring to each public (visible to users): - - * module - * function - * class - * method - -\When you add a new module, you should include a docstring with a short sentence describing what the module does, and/or a long document including examples and references. - -.. _guidelines-for-docstrings: - -Guidelines for writing docstrings ---------------------------------- - -A typical function docstring looks like the following: - - :: - - def func(arg1, arg2): - """Summary line. - - Extended description of function. - - Parameters - ---------- - arg1 : int - Description of `arg1` - arg2 : str - Description of `arg2` - - - Returns - ------- - bool - Description of return value - - """ - return True - -.. seealso:: - - The `napoleon documentation `_ has further breakdowns of docstrings at the module, function, class, method, variable, and other levels. - -* When writing reST markup, make sure that there are **at least two blank lines above** the reST after a numpy heading. Otherwise, the Sphinx/napoleon parser does not render correctly. - - .. code-block:: RST - - some more docs bla bla - - Notes - ----- - THE NEXT TWO BLANK LINES ARE IMPORTANT. - - - .. versionadded:: 0.16.0 - -* Do not use "Example" or "Examples" as a normal section heading (e.g. in module level docs): *only* use it as a `NumPy doc Section `__. It will not be rendered properly, and will mess up sectioning. - - -* When writing multiple common names in one line, Sphinx sometimes tries to reference the first name. In that case, you have to split the names across multiple lines. See below for an example: - - .. code-block:: RST - - Parameters - ---------- - n_atoms, n_residues : int - numbers of atoms/residues - -* We are using MathJax with sphinx so you can write LaTeX code in math tags. - - In blocks, the code below - - .. code-block:: rst - - # - .. math:: - e^{i\pi} = -1 - - renders like so: - - .. math:: - e^{i\pi} = -1 - - - Math directives can also be used inline. - - .. code-block:: rst - - We make use of the identity :math:`e^{i\pi} = -1` to show... - - Note that you should *always* make doc strings with math code **raw** python strings **by prefixing them with the letter "r"**, or else you will get problems with backslashes in unexpected places. - - :: - - def rotate(self, R): - r"""Apply a rotation matrix *R* to the selection's coordinates. - - :math:`\mathsf{R}` is a 3x3 orthogonal matrix that transforms a vector - :math:`\mathbf{x} \rightarrow \mathbf{x}'`: - - .. math:: - - \mathbf{x}' = \mathsf{R}\mathbf{x} - """ - - .. seealso:: - - See `Stackoverflow: Mathjax expression in sphinx python not rendering correctly `_ for further discussion. - - -Writing docs for abstract base classes --------------------------------------- - -MDAnalysis contains a number of abstract base classes, such as :class:`~MDAnalysis.analysis.base.AnalysisBase`. Developers who define new base classes, or modify existing ones, should follow these rules: - - - The *class docstring* needs to contain a list of methods that can be overwritten by inheritance from the base class. Distinguish and document methods as required or optional. - - The class docstring should contain a minimal example for how to derive this class. This demonstrates best practices, documents ideas and intentions behind the specific choices in the API, helps to promote a unified code base, and is useful for developers as a concise summary of the API. - - A more detailed description of methods should come in the *method docstring*, with a note specifying if the method is required or optional to overwrite. - -See the documentation of :class:`MDAnalysis.analysis.base.AnalysisBase` for an example of this documentation. - -Adding your documentation to MDAnalysis ---------------------------------------- - -As with any contribution to an MDAnalysis repository, :ref:`commit and push ` your documentation contributions to GitHub. If *any fixes in the restructured text* are needed, *put them in their own commit* (and do not include any generated files under `docs/html`). Try to keep all reST fixes in the one commit. ``git add FILE`` and ``git commit --amend`` is your friend when piling more and more small reST fixes onto a single "fixed reST" commit. - -Then, :ref:`create a pull request `. All the tests in the MDAnalysis test suite will run, but only one checks that the documents compile correctly. - -Viewing the documentation interactively ---------------------------------------- - -In the Python interpreter one can simply say: - - :: - - import MDAnalysis - help(MDAnalysis) - help(MDAnalysis.Universe) - -In ``ipython`` one can use the question mark operator: - - .. ipython:: - :verbatim: - - In [1]: MDAnalysis.Universe? - -.. _autobuild-sphinx: - ------------------------------------- -Automatically building documentation ------------------------------------- - -Constantly rebuilding documentation can become tedious when you have many changes to make. Use `sphinx-autobuild `_ to rebuild documentation every time you make changes to any document, including Jupyter notebooks. Install ``sphinx-autobuild``: - - .. code-block:: bash - - pip install sphinx-autobuild - -Then, run the following command in the ``doc/`` directory: - - .. code-block:: bash - - python -m sphinx_autobuild source build - -This will start a local webserver at http://localhost:8000/, which will refresh every time you save changes to a file in the documentation. This is helpful for both the user guide (first navigate to ``UserGuide/doc``) and the main repository documentation (navigate to ``package/doc/sphinx``). - - - -.. _Issue Tracker: https://github.com/MDAnalysis/mdanalysis/issues -.. _`pull request`: https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests .. _`mdnalysis-discussion`: http://groups.google.com/group/mdnalysis-discussion .. _`Code of Conduct`: https://www.mdanalysis.org/pages/conduct/ -.. _`Git`: http://git-scm.com/ -.. _`PEP8`: https://www.python.org/dev/peps/pep-0008/ -.. _`flake8`: http://flake8.readthedocs.org/en/latest/ -.. _`PyCharm`: https://www.jetbrains.com/pycharm/ -.. _`elpy`: https://github.com/jorgenschaefer/elpy -.. _`python-mode`: https://github.com/klen/python-mode -.. _`autopep8`: https://github.com/hhatto/autopep8 -.. _`yapf`: https://github.com/google/yapf -.. _`Sphinx`: http://sphinx.pocoo.org/ +.. _`Issue Tracker`: https://github.com/MDAnalysis/mdanalysis/issues diff --git a/doc/source/contributing_code.rst b/doc/source/contributing_code.rst new file mode 100644 index 000000000..2bec643aa --- /dev/null +++ b/doc/source/contributing_code.rst @@ -0,0 +1,721 @@ +.. _working-with-mdanalysis-code: + +================================= +Contributing to the main codebase +================================= + +If you would like to contribute, start by searching through the `issues `_ and :ref:`pull requests ` to see whether someone else has raised a similar idea or question. + +If you don't see your idea or problem listed, do one of the following: + + * If your contribution is **minor**, such as a typo fix, go ahead and fix it by following the guide below and :ref:`open a pull request `. + + * If your contribution is **major**, such as a bug fix or a new feature, start by opening an issue first. That way, other people can weigh in on the discussion before you do any work. If you also create a pull request, you should link it to the issue by including the issue number in the pull request’s description. + +Here is an overview of the development workflow for code or inline code documentation, as expanded on throughout the rest of the page. + + #. :ref:`Fork the MDAnalysis repository ` from the mdanalysis account into your own account + #. :ref:`Set up an isolated virtual environment ` for code development + #. :ref:`Build development versions ` of MDAnalysis and MDAnalysisTests on your computer into the virtual environment + #. :ref:`Create a new branch off the develop branch ` + #. :ref:`Add your new feature or bug fix ` or :ref:`add your new documentation ` + #. :ref:`Add and run tests ` (if adding to the code) + #. :ref:`Build and view the documentation ` (if adding to the docs) + #. :ref:`Commit and push your changes, and open a pull request. ` + + +Working with the code +===================== + +.. _forking-code-repo: + +------- +Forking +------- + +You will need your `own fork `_ to work on the code. Go to the `MDAnalysis project page `_ and hit the :guilabel:`Fork` button. You will +want to `clone your fork `_ to your machine: + +.. code-block:: bash + + git clone https://github.com/your-user-name/mdanalysis.git + cd mdanalysis + git remote add upstream https://github.com/MDAnalysis/mdanalysis + +This creates the directory `mdanalysis` and connects your repository to +the upstream (main project) MDAnalysis repository. + +.. _create-virtual-environment: + +---------------------------------- +Creating a development environment +---------------------------------- + +To change code and test changes, you'll need to build both **MDAnalysis** and **MDAnalysisTests** +from source. This requires a Python environment. We highly recommend that you use +virtual environments. This allows you to have multiple experimental development versions +of MDAnalysis that do not interfere with each other, or your own stable version. +Since MDAnalysis is split into the actual package and a test suite, you need to install +both modules in development mode. + +You can do this either with :ref:`conda ` or :ref:`pip `. + +.. _dev-with-conda: + +With conda +---------- + +Install either `Anaconda `_ +or `miniconda `_. +Make sure your conda is up to date: + + .. code-block:: bash + + conda update conda + +Create a new environment with ``conda create``. This will allow you to change code in +an isolated environment without touching your base Python installation, and without +touching existing environments that may have stable versions of MDAnalysis. : + + .. code-block:: bash + + conda create --name mdanalysis-dev + +Activate the environment to build MDAnalysis into it: + + .. code-block:: bash + + conda activate mdanalysis-dev + +To view your environments: + + .. code-block:: bash + + conda info -e + +To list the packages installed in your current environment: + + .. code-block:: bash + + conda list + +To return to your root environment: + + .. code-block:: bash + + conda deactivate + +See the full conda docs `here `__. + +.. _dev-with-pip: + +With pip and virtualenv +----------------------- + +Like conda, virtual environments managed with `virtualenv `_ allow you to use different versions of Python and Python packages for your different project. Unlike conda, virtualenv is not a general-purpose package manager. Instead, it leverages what is available on your system, and lets you install Python packages using pip. + +To use virtual environments you have to install the virtualenv package first. This can be done with either pip or the package manager of your system: + + .. code-block:: bash + + pip install virtualenv + # or on ubuntu + sudo apt install virtualenv + # or on fedora + sudo dnf install python-virtualenv + +Virtual environments can be created for each project directory. + + .. code-block:: bash + + cd my-project/ + virtualenv my-project-env + +This will create a new folder ``my-project-env``. This folder contains the virtual environment and all packages you have installed in it. To activate it in the current terminal run: + + .. code-block:: bash + + source myproject-env/bin/activate + +Now you can install packages via pip without affecting your global environment. The packages that you install when the environment is activated will be available in terminal sessions that have the environment activated. You can deactivate the virtual environment by running: + + .. code-block:: bash + + deactivate + +The `virtualenvwrapper package `_ makes virtual environments easier to use. It provides some very useful features: + + - it organises the virtual environment into a single user-defined directory, so they are not scattered throughout the file system; + - it defines commands for the easy creation, deletion, and copying of virtual environments; + - it defines a command to activate a virtual environment using its name; + - all commands defined by ``virtualenvwrapper`` have tab-completion for virtual environment names. + +You first need to install ``virtualenvwrapper`` *outside* of a virtual environment: + + .. code-block:: bash + + pip install virtualenvwrapper + # or on ubuntu + sudo apt install virtualenvwrapper + # or on fedora + sudo dnf install python-virtualenvwrapper + +Then, you need to load it into your terminal session. Add the following lines in ``~/.bashrc``. They will be executed every time you open a new terminal session: + + .. code-block:: bash + + # Decide where to store the virtual environments + export WORKON_HOME=~/Envs + # Make sure the directory exists + mkdir -p ${WORKON_HOME} + # Load virtualenvwrapper + source /usr/local/bin/virtualenvwrapper.sh + +Open a new terminal or run ``source ~/.bashrc`` to update your session. You can now create a virtual environment with: + + .. code-block:: bash + + mkvirtualenv my-project + +Regardless of your current working directory, the environment is created in ``~/Envs/`` and it is now loaded in our terminal session. + +You can load your virtual environments by running ``workon my-project``, and exit them by running ``deactivate``. + +Virtual environments, especially with ``virtualenvwrapper``, can do much more. For example, you can create virtual environments with different python interpreters with the ``-p`` flag. The Hitchhiker's Guide to Python has a good `tutorial `_ that gives a more in-depth explanation of virtual environments. The `virtualenvwrapper documentation `_ is also a good resource to read. + +On a Mac +-------- + +One more step is often required on macOS, because of the default number of files that a process can open simultaneously is quite low (256). To increase the number of files that can be accessed, run the following command: + + .. code-block:: bash + + ulimit -n 4096 + +This sets the number of files to 4096. However, this command only applies to your currently open terminal session. To keep this high limit, add the above line to your ``~/.profile``. + + + +.. _build-mdanalysis-develop: + +------------------- +Building MDAnalysis +------------------- + +Make sure that you have :ref:`cloned the repository ` +and activated your virtual environment. First we need to install dependencies: + + .. code-block:: bash + + # if using conda + conda install -c biobuilds -c conda-forge \ + cython numpy mmtf-python mock six biopython \ + networkx cython matplotlib scipy griddataformats \ + hypothesis gsd codecov "seaborn>=0.7.0,<=0.9" \ + clustalw=2.1 netcdf4 scikit-learn "joblib>=0.12" + # if using pip + pip install cython numpy mmtf-python mock six biopython \ + networkx cython matplotlib scipy griddataformats \ + hypothesis gsd codecov "seaborn>=0.7.0,<=0.9" \ + netcdf4 scikit-learn "joblib>=0.12" + +Ensure that you have a working C/C++ compiler (e.g. gcc or clang). You will also need Python ≥ 3.4. We will now install MDAnalysis. + + .. code-block:: bash + + # go to the mdanalysis source directory + cd mdanalysis/ + + # Build and install the MDAnalysis package + cd package/ + pip install -e . + + # Build and install the test suite + cd ../testsuite/ + pip install -e . + +At this point you should be able to import MDAnalysis from your locally built version, this is visible from the version number ending in "-dev0" for example: + + .. code-block:: bash + + $ python # start an interpreter + >>> import MDAnalysis as mda + >>> mda.__version__ + '0.20.2-dev0' + +.. _branches-in-mdanalysis: + +---------------------- +Branches in MDAnalysis +---------------------- + +There are two important branches in MDAnalysis: + + - ``master``: for production-ready code + - ``develop``: for development code + +The ``master`` branch is only for stable, production-ready code. Development code should *never* be committed to this branch. Typically, code is only committed by the release manager, when a release is ready. + +The ``develop`` branch can be considered an "integration" branch for including your code into the next release. Only working, tested code should be committed to this branch. Code contributions ("features") should branch off ``develop`` rather than ``master``. + + +.. _create-code-branch: + +Creating a branch +----------------- + +The develop branch should only contain approved, tested code, so create a +feature branch for making your changes. For example, to create a branch called +``shiny-new-feature`` from ``develop``: + + .. code-block:: bash + + git checkout -b shiny-new-feature develop + +This changes your working directory to the ``shiny-new-feature`` branch. Keep any +changes in this branch specific to one bug or feature so it is clear +what the branch brings to MDAnalysis. You can have many branches with different names +and switch in between them using the ``git checkout my-branch-name`` command. + +There are several special branch names that you should not use for your feature branches: + + - ``master`` + - ``develop`` + - ``release-*`` + + +``release`` branches are used to :ref:`prepare a new production release ` and should be handled by the release manager only. + +.. _writing-new-code: + +---------------- +Writing new code +---------------- + +Code formatting in Python +------------------------- + +MDAnalysis is a project with a long history and many contributors; it hasn't used a consistent coding style. Since version 0.11.0, we are trying to update all the code to conform with `PEP8`_. Our strategy is to update the style every time we touch an old function and thus switch to `PEP8`_ continuously. + +**Important requirements (from PEP8):** + - keep line length to **79 characters or less**; break long lines sensibly + - indent with **spaces** and use **4 spaces per level** + - naming: + + - classes: `CapitalClasses` (i.e. capitalized nouns without spaces) + - methods and functions: `underscore_methods` (lower case, with underscores for spaces) + +We recommend that you use a Python Integrated Development Environment (IDE) (`PyCharm`_ and others) or external tools like `flake8`_ for code linting. For integration of external tools with emacs and vim, check out `elpy`_ (emacs) and `python-mode`_ (vim). + +To apply the code formatting in an automated way, you can also use code formatters. External tools include `autopep8`_ and `yapf`_. Most IDEs either have their own code formatter or will work with one of the above through plugins. + + +.. _`PEP8`: https://www.python.org/dev/peps/pep-0008/ +.. _`flake8`: http://flake8.readthedocs.org/en/latest/ +.. _`PyCharm`: https://www.jetbrains.com/pycharm/ +.. _`elpy`: https://github.com/jorgenschaefer/elpy +.. _`python-mode`: https://github.com/klen/python-mode +.. _`autopep8`: https://github.com/hhatto/autopep8 +.. _`yapf`: https://github.com/google/yapf + + +Modules and dependencies +------------------------ + +MDAnalysis strives to keep dependencies small and lightweight. Code outside the :mod:`MDAnalysis.analysis` and :mod:`MDAnalysis.visualization` modules should only rely on the :ref:`core dependencies `, which are always installed. Analysis and visualization modules can use any :ref:`any package, but the package is treated as optional `. + +Imports in the code should follow the :ref:`general-rules-for-importing`. + +.. seealso:: + + See :ref:`module-imports` for more information. + + +Developing in Cython +-------------------- + +The ``setup.py`` script first looks for the `.c` files included in the standard MDAnalysis distribution. These are not in the GitHub repository, so ``setup.py`` will use Cython to compile extensions. `.pyx` source files are used instead of `.c` files. From there, `.pyx` files are converted to `.c` files if they are newer than the already present `.c` files or if the ``--force`` flag is set (i.e. ``python setup.py build --force``). End users (or developers) should not trigger the `.pyx` to `.c` conversion, since `.c` files delivered with source packages are always up-to-date. However, developers who work on the `.pyx` files will automatically trigger the conversion since `.c` files will then be outdated. + +Place all source files for compiled shared object files into the same directory as the final shared object file. + +`.pyx` files and cython-generated `.c` files should be in the same directory as the `.so` files. External dependent C/C++/Fortran libraries should be in dedicated ``src/`` and ``include/`` folders. See the following tree as an example: + + :: + + MDAnalysis + |--lib + | |-- _distances.so + | |-- distances.pyx + | |-- distances.c + |-- coordinates + |-- _dcdmodule.so + |-- src + |-- dcd.c + |-- include + |-- dcd.h + +.. _test-code: + +----------------- +Testing your code +----------------- + +MDAnalysis takes testing seriously. All code added to MDAnalysis should have tests to ensure that it works as expected; we aim for 90% coverage. See :ref:`testing` for more on :ref:`writing `, :ref:`running `, and interpreting tests. + + +--------------------- +Documenting your code +--------------------- + +Changes to the code should be reflected in the ongoing ``CHANGELOG``. Add an entry here to document your fix, enhancement, or change. In addition, add your name to the author list. If you are addressing an issue, make sure to include the issue number. + + +.. _adding-code-to-mda: + +------------------------------ +Adding your code to MDAnalysis +------------------------------ + +Committing your code +-------------------- + +When you are happy with a set of changes and :ref:`all the tests pass `, it is time to commit. All changes in one revision should have a common theme. If you implemented two rather different things (say, one bug fix and one new feature), then split them into two commits with different messages. + +Once you’ve made changes to files in your local repository, you can see them by typing: + + .. code-block:: bash + + git status + +Tell git to track files by typing: + + .. code-block:: + + git add path/to/file-to-be-added.py + +Doing ``git status`` again should give something like: + + .. code-block:: + + # On branch shiny-new-feature + # + # modified: /relative/path/to/file-you-added.py + # + +Then commit with: + + .. code-block:: bash + + git commit -m + +This opens up a message editor. + +*Always* add a descriptive comment for your commit message (feel free to be verbose!): + + - use a short (<50 characters) subject line that summarizes the change + - leave a blank line + - optionally, add additional more verbose descriptions; paragraphs or bullet lists (with - or *) are good + - manually break lines at 80 characters + - manually indent bullet lists + +.. seealso:: + + See `Tim Pope's A Note About Git Commit Messages `_ for a rationale for these rules. + + +Pushing your code to GitHub +--------------------------- + +When you want your changes to appear publicly on your GitHub page, push your forked feature branch’s commits: + + .. code-block:: bash + + git push origin shiny-new-feature + +Here `origin` is the default name given to your remote repository on GitHub. You can see the remote repositories: + + .. code-block:: bash + + git remote -v + +If you added the upstream repository as described above you will see something like: + + .. code-block:: bash + + origin git@github.com:your-username/mdanalysis.git (fetch) + origin git@github.com:your-username/mdanalysis.git (push) + upstream git@github.com:MDAnalysis/mdanalysis.git (fetch) + upstream git@github.com:MDAnalysis/mdanalysis.git (push) + +Now your code is on GitHub, but it is not yet a part of the MDAnalysis project. For that to happen, a pull request needs to be submitted on GitHub. + +.. _rebase-code: + +Rebasing your code +------------------ + +Often the upstream MDAnalysis develop branch will be updated while you are working on your own code. +You will then need to update your own branch with the new code to avoid merge conflicts. +You need to first retrieve it and then `rebase `_ +your branch so that your changes apply to the new code: + + .. code-block:: bash + + git fetch upstream + git rebase upstream/develop + +This will replay your commits on top of the latest development code from MDAnalysis. If this +leads to merge conflicts, you must resolve these before submitting your pull +request. If you have uncommitted changes, you will need to ``git stash`` them +prior to updating. This will effectively store your changes and they can be +reapplied after updating with ``git stash apply``. + +Once rebased, push your changes: + + .. code-block:: bash + + git push -f origin shiny-new-feature + +and `create a pull request `_. + +.. _create-a-pull-request: + +Creating a pull request +----------------------- + +The typical approach to adding your code to MDAnalysis is to make a `pull request `_ on GitHub. Please make sure that your contribution :ref:`passes all tests `. If there are test failures, you will need to address them before we can review your contribution and eventually merge them. If you have problems with making the tests pass, please ask for help! (You can do this in the comments of the pull request). + + #. Navigate to your repository on GitHub + #. Click on the :guilabel:`Pull Request` button + #. You can then click on :guilabel:`Commits` and :guilabel:`Files Changed` to make sure everything looks okay one last time + #. Write a description of your changes and follow the PR checklist + + - check that docs are updated + - check that tests run + - check that you've updated CHANGELOG + - reference the issue that you address, if any + + #. Click :guilabel:`Send Pull Request`. + +Your pull request is then sent to the repository maintainers. After this, the following happens: + + #. A :ref:`suite of tests are run on your code ` with the tools :ref:`travis`, :ref:`appveyor` and :ref:`codecov`. If they fail, please fix your pull request by pushing updates to it. + #. Developers will ask questions and comment in the pull request. You may be asked to make changes. + #. When everything looks good, a core developer will merge your code into the ``develop`` branch of MDAnalysis. Your code will be in the next release. + +If you need to make changes to your code, you can do so on your local repository as you did before. Committing and pushing the changes will update your pull request and restart the automated tests. + +.. _working-with-mdanalysis-docs: + +Working with the code documentation +=================================== + +MDAnalysis maintains two kinds of documentation: + + #. `This user guide `__: a map of how MDAnalysis works, combined with tutorial-like overviews of specific topics (such as the analyses) + + #. `The documentation generated from the code itself `__. Largely built from code docstrings, these are meant to provide a clear explanation of the usage of individual classes and functions. They often include technical or historical information such as in which version the function was added, or deprecation notices. + +This guide is for the documentation generated from the code. If you are looking to contribute to the user guide, please see :ref:`working-with-user-guide`. + +MDAnalysis has a lot of documentation in the Python doc strings. The docstrings follow the `Numpy Docstring Standard `__, which is used widely +in the Scientific Python community. They are nice to read as normal text and are converted by sphinx to normal ReST through `napoleon `__. + +This standard specifies the format of +the different sections of the docstring. See `this document +`_ +for a detailed explanation, or look at some of the existing functions to +extend it in a similar manner. + +Note that each page of the `online documentation `_ has a link to the *Source* of the page. You can look at it in order to find out how a particular page has been written in reST and copy the approach for your own documentation. + +.. _building-code-documentation: + +-------------------------- +Building the documentation +-------------------------- + +The online documentation is generated from the pages in ``mdanalysis/package/doc/sphinx/source/documentation_pages``. The documentation for the current release are hosted at www.mdanalysis.org/docs, while the development version is at www.mdanalysis.org/mdanalysis/. + +In order to build the documentation, you must first :ref:`clone the main MDAnalysis repo `. :ref:`Set up a virtual environment ` in the same way as you would for the code (you can use the same environment as you do for the code). You will need to install several packages for the docs. + + .. code-block:: bash + + pip install sphinx sphinx-sitemap sphinx_rtd_theme + +In addition, build the development version of MDAnalysis (if you haven't done this already): + + .. code-block:: bash + + pip install -e . + +Then, generate the docs with: + + .. code-block:: bash + + python setup.py build_sphinx -E + +This generates and updates the files in ``doc/html``. If the above command fails with an ``ImportError``, run + + .. code-block:: bash + + python setup.py build_ext --inplace + +and retry. + +You will then be able to open the home page, ``doc/html/index.html``, and look through the docs. In particular, have a look at any pages that you tinkered with. It is typical to go through multiple cycles of fix, rebuild the docs, check and fix again. + +If rebuilding the documentation becomes tedious after a while, install the :ref:`sphinx-autobuild ` extension. + +------------------------- +Where to write docstrings +------------------------- + +When writing Python code, you should always add a docstring to each public (visible to users): + + * module + * function + * class + * method + +\When you add a new module, you should include a docstring with a short sentence describing what the module does, and/or a long document including examples and references. + +.. _guidelines-for-docstrings: + +--------------------------------- +Guidelines for writing docstrings +--------------------------------- + +A typical function docstring looks like the following: + + :: + + def func(arg1, arg2): + """Summary line. + + Extended description of function. + + Parameters + ---------- + arg1 : int + Description of `arg1` + arg2 : str + Description of `arg2` + + + Returns + ------- + bool + Description of return value + + """ + return True + +.. seealso:: + + The `napoleon documentation `_ has further breakdowns of docstrings at the module, function, class, method, variable, and other levels. + +* When writing reST markup, make sure that there are **at least two blank lines above** the reST after a numpy heading. Otherwise, the Sphinx/napoleon parser does not render correctly. + + .. code-block:: RST + + some more docs bla bla + + Notes + ----- + THE NEXT TWO BLANK LINES ARE IMPORTANT. + + + .. versionadded:: 0.16.0 + +* Do not use "Example" or "Examples" as a normal section heading (e.g. in module level docs): *only* use it as a `NumPy doc Section `__. It will not be rendered properly, and will mess up sectioning. + + +* When writing multiple common names in one line, Sphinx sometimes tries to reference the first name. In that case, you have to split the names across multiple lines. See below for an example: + + .. code-block:: RST + + Parameters + ---------- + n_atoms, n_residues : int + numbers of atoms/residues + +* We are using MathJax with sphinx so you can write LaTeX code in math tags. + + In blocks, the code below + + .. code-block:: rst + + # + .. math:: + e^{i\pi} = -1 + + renders like so: + + .. math:: + e^{i\pi} = -1 + + + Math directives can also be used inline. + + .. code-block:: rst + + We make use of the identity :math:`e^{i\pi} = -1` to show... + + Note that you should *always* make doc strings with math code **raw** python strings **by prefixing them with the letter "r"**, or else you will get problems with backslashes in unexpected places. + + :: + + def rotate(self, R): + r"""Apply a rotation matrix *R* to the selection's coordinates. + + :math:`\mathsf{R}` is a 3x3 orthogonal matrix that transforms a vector + :math:`\mathbf{x} \rightarrow \mathbf{x}'`: + + .. math:: + + \mathbf{x}' = \mathsf{R}\mathbf{x} + """ + + .. seealso:: + + See `Stackoverflow: Mathjax expression in sphinx python not rendering correctly `_ for further discussion. + +-------------------------------------- +Writing docs for abstract base classes +-------------------------------------- + +MDAnalysis contains a number of abstract base classes, such as :class:`~MDAnalysis.analysis.base.AnalysisBase`. Developers who define new base classes, or modify existing ones, should follow these rules: + + - The *class docstring* needs to contain a list of methods that can be overwritten by inheritance from the base class. Distinguish and document methods as required or optional. + - The class docstring should contain a minimal example for how to derive this class. This demonstrates best practices, documents ideas and intentions behind the specific choices in the API, helps to promote a unified code base, and is useful for developers as a concise summary of the API. + - A more detailed description of methods should come in the *method docstring*, with a note specifying if the method is required or optional to overwrite. + +See the documentation of :class:`MDAnalysis.analysis.base.AnalysisBase` for an example of this documentation. + +--------------------------------------- +Adding your documentation to MDAnalysis +--------------------------------------- + +As with any contribution to an MDAnalysis repository, :ref:`commit and push ` your documentation contributions to GitHub. If *any fixes in the restructured text* are needed, *put them in their own commit* (and do not include any generated files under `docs/html`). Try to keep all reST fixes in the one commit. ``git add FILE`` and ``git commit --amend`` is your friend when piling more and more small reST fixes onto a single "fixed reST" commit. + +We recommend :ref:`building the docs locally first ` to preview your changes. Then, :ref:`create a pull request `. All the tests in the MDAnalysis test suite will run, but only one checks that the documents compile correctly. + +--------------------------------------- +Viewing the documentation interactively +--------------------------------------- + +In the Python interpreter one can simply say: + + :: + + import MDAnalysis + help(MDAnalysis) + help(MDAnalysis.Universe) + +In ``ipython`` one can use the question mark operator: + + .. ipython:: + :verbatim: + + In [1]: MDAnalysis.Universe? diff --git a/doc/source/contributing_docs.rst b/doc/source/contributing_docs.rst new file mode 100644 index 000000000..6a68b9d21 --- /dev/null +++ b/doc/source/contributing_docs.rst @@ -0,0 +1,144 @@ + +.. _working-with-user-guide: + +============================== +Contributing to the user guide +============================== + +MDAnalysis maintains two kinds of documentation: + + #. `This user guide `__: a map of how MDAnalysis works, combined with tutorial-like overviews of specific topics (such as the analyses) + + #. `The documentation generated from the code itself `__. Largely built from code docstrings, these are meant to provide a clear explanation of the usage of individual classes and functions. They often include technical or historical information such as in which version the function was added, or deprecation notices. + +This guide is about how to contribute to the user guide. If you are looking to add to documentation of the main code base, please see :ref:`working-with-mdanalysis-docs`. + +The user guide makes use of a number of Sphinx extensions to ensure that the code examples are always up-to-date. These include `nbsphinx `_ and the `ipython directive `__. + +The ``ipython`` directive lets you put code in the documentation which will be run +during the doc build. For example: + + :: + + .. ipython:: python + + x = 2 + x**3 + +will be rendered as: + + .. ipython:: + + In [1]: x = 2 + + In [2]: x**3 + Out[2]: 8 + +Many code examples in the docs are run during the +doc build. This approach means that code examples will always be up to date, +but it does make the doc building a bit more complex. + +Here is an overview of the development workflow for the user guide, as expanded on throughout the rest of the page. + + #. :ref:`Fork the MDAnalysis repository ` from the mdanalysis account into your own account + #. :ref:`Set up an isolated virtual environment ` for your documentation + #. :ref:`Create a new branch off the master branch ` + #. Add your new documentation. + #. :ref:`Build and view the documentation `. + #. :ref:`Commit and push your changes, and open a pull request `. + + +.. _forking-user-guide: + +Forking and cloning the User Guide +================================== + +Go to the `MDAnalysis project page `_ and hit the :guilabel:`Fork` button. You will +want to clone your fork to your machine: + + .. code-block:: bash + + git clone https://github.com/your-user-name/UserGuide.git + cd UserGuide + git remote add upstream https://github.com/MDAnalysis/UserGuide + +This creates the directory `UserGuide` and connects your repository to +the upstream (main project) MDAnalysis repository. + + +.. _create-virtual-environment-user-guide: + +Creating a development environment +================================== + +:ref:`Create a new virtual environment ` for the user guide. Install the required dependencies, and activate the ``nglview`` extension. We use ``nglview`` for visualizing molecules in Jupyter notebook tutorials. + +If using conda: + + .. code-block:: bash + + cd UserGuide/ + conda env create python=3.6 -f environment.yml --quiet + conda activate mda-user-guide + jupyter-nbextension enable nglview --py --sys-prefix + + +If using pip: + + .. code-block:: bash + + cd UserGuide/ + pip install -r requirements.txt + jupyter-nbextension enable nglview --py --sys-prefix + +.. _build-user-guide: + +Building the user guide +======================= + +Navigate to the ``doc/`` directory and run ``make html``: + + .. code-block:: bash + + cd doc/ + make html + +The HTML output will be in ``doc/build/``, which you can open in your browser of choice. The homepage is ``doc/build/index.html``. + +If rebuilding the documentation becomes tedious after a while, install the :ref:`sphinx-autobuild ` extension. + +Saving state in Jupyter notebooks +================================= + +One of the neat things about ``nglview`` is the ability to interact with molecules via the viewer. This ability can be preserved for the HTML pages generated from Jupyer notebooks by ``nbsphinx``, if you save the notebook widget state after execution. + +.. _add-changes-user-guide: + +Adding changes to the UserGuide +=============================== + +As with the code, :ref:`commit and push ` your code to GitHub. Then :ref:`create a pull request `. The only test run for the User Guide is: that your file compile into HTML documents without errors. As usual, a developer will review your PR and merge the code into the User Guide when it looks good. + + +.. _autobuild-sphinx: + +Automatically building documentation +==================================== + +Constantly rebuilding documentation can become tedious when you have many changes to make. Use `sphinx-autobuild `_ to rebuild documentation every time you make changes to any document, including Jupyter notebooks. Install ``sphinx-autobuild``: + + .. code-block:: bash + + pip install sphinx-autobuild + +Then, run the following command in the ``doc/`` directory: + + .. code-block:: bash + + python -m sphinx_autobuild source build + +This will start a local webserver at http://localhost:8000/, which will refresh every time you save changes to a file in the documentation. This is helpful for both the user guide (first navigate to ``UserGuide/doc``) and the main repository documentation (navigate to ``package/doc/sphinx``). + + + + diff --git a/doc/source/datasets.rst b/doc/source/datasets.rst new file mode 100644 index 000000000..6ebbc7a18 --- /dev/null +++ b/doc/source/datasets.rst @@ -0,0 +1,69 @@ +.. -*- coding: utf-8 -*- +.. _datasets: + +============ +Example data +============ + +MDAnalysis offers a collection of example data files and datasets to run tests and tutorials. These are split into two packages: + + * MDAnalysisTests: primarily for unit tests of the code + * MDAnalysisData: datasets for workshops and tutorials + +.. _mdanalysistests: + +MDAnalysisTests +=============== + +While this is installed as a separate package, you should import files like so:: + + import MDAnalysis as mda + from MDAnalysis.tests.datafiles import PSF, DCD + + u = mda.Universe(PSF, DCD) + +A complete list of files and descriptions is in the ``mdanalysis/testsuite/MDAnalysisTests/datafiles.py`` file. The actual files are stored in the ``mdanalysis/testsuite/MDAnalysisTests/data/`` directory. + +.. _mdanalysisdata: + +MDAnalysisData +============== + +The `MDAnalysisData `_ package is an interface to download, cache, and access certain datasets hosted on external repositories (e.g. figshare_, zenodo_, DataDryad_). Data is not downloaded upon installation, so the package itself is small; but the directory where the datasets are cached can grow significantly. + +You can access datasets like so:: + + import MDAnalysis as mda + from MDAnalysisData import datasets + adk = datasets.fetch_adk_equilibrium() + u = mda.Universe(adk.topology, adk.trajectory) + + # to see the description of the dataset + print(adk.DESCR) + +The cached files are stored by default in the ``~/MDAnalysis_data`` directory. This can be changed by setting the environment variable ``MDANALYSIS_DATA``. You can change this for a Terminal session with the below command: + +.. code-block:: bash + + export MDANALYSIS_DATA=/my/chosen/path/MDAnalysis_data + +Add it to your ``.bashrc`` for a permanent change. + +.. code-block:: bash + + echo 'export MDANALYSIS_DATA=/my/chosen/path/MDAnalysis_data' >> ~/.bashrc + +In Python, you can check the location of your caching directory:: + + MDAnalysisData.base.get_data_home() + +And clear the directory with:: + + MDAnalysisData.base.clear_data_home() + +`A list of datasets `_ can be found at the MDAnalysisData documentation. + + +.. _figshare: https://figshare.com/ +.. _zenodo: https://zenodo.org/ +.. _DataDryad: https://www.datadryad.org/ \ No newline at end of file diff --git a/doc/source/examples/analysis/polymers_and_membranes/sphpdb.png b/doc/source/examples/analysis/polymers_and_membranes/sphpdb.png new file mode 100644 index 000000000..bc8b38e9a Binary files /dev/null and b/doc/source/examples/analysis/polymers_and_membranes/sphpdb.png differ diff --git a/doc/source/faq.rst b/doc/source/faq.rst index 201e3165c..62c4d69f3 100644 --- a/doc/source/faq.rst +++ b/doc/source/faq.rst @@ -17,14 +17,5 @@ only one time frame of the trajectory is being accessed. The Think of the trajectory as a function :math:`X(t)` of the frame index :math:`t` that makes the data from this specific frame available. This structure is important because it allows MDAnalysis to work with trajectory files too large to fit -into the computer's memory. +into the computer's memory. See :ref:`trajectories` for more information. -The Timestep data structure :code:`my_universe.trajectory.ts` holds frame information. -You also see the Timestep when you iterate through a trajectory: - -.. code-block:: python - - for ts in my_universe.trajectory: - print(ts.frame, ts.time) - -This prints the frame index (starting with 0) and the time for all frames. diff --git a/doc/source/index.rst b/doc/source/index.rst index aa44c3dd7..466cea7b8 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -3,6 +3,7 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. +================================================= Welcome to MDAnalysis User Guide's documentation! ================================================= @@ -10,29 +11,63 @@ Welcome to MDAnalysis User Guide's documentation! **Last updated:** |today| -**MDAnalysis** (`www.mdanalysis.org`_) is an object-oriented python -toolkit to analyze molecular dynamics files and trajectories generated by -CHARMM_, Gromacs_, Amber_, NAMD_, LAMMPS_, `DL_POLY`_ and other -packages (see :ref:`formats` for full information). MDAnalysis can write -most of these formats, too, together with atom selections for use in -Gromacs_, CHARMM_, VMD_ and PyMol_ (see :ref:`selection-exporters`). It uses Numpy_ arrays to provide a flexible and relatively fast framework for complex analysis tasks. +**MDAnalysis** (`www.mdanalysis.org`_) is a Python +toolkit to analyse molecular dynamics files and trajectories in :ref:`many popular formats `. MDAnalysis can write +most of these formats, too, together with atom selections for use in :ref:`visualisation tools or other analysis programs `. +It provides a fast framework for :ref:`complex analysis tasks `, +as well as flexible tooling to construct your own analyses. .. _`www.mdanalysis.org`: https://www.mdanalysis.org -.. _NumPy: http://numpy.scipy.org -.. _CHARMM: http://www.charmm.org/ -.. _Amber: http://ambermd.org/ -.. _LAMMPS: http://lammps.sandia.gov/ -.. _NAMD: http://www.ks.uiuc.edu/Research/namd/ -.. _Gromacs: http://www.gromacs.org/ -.. _`DL_POLY`: http://www.scd.stfc.ac.uk//44516.aspx -.. _VMD: http://www.ks.uiuc.edu/Research/vmd/ -.. _PyMol: http://www.pymol.org/ -.. _PDB: http://www.rcsb.org/pdb/static.do?p=file_formats/pdb/index.html -.. _XYZ format: http://openbabel.org/wiki/XYZ_%28format%29 + +Why MDAnalysis? +=============== + +The typical use case for MDAnalysis is to manipulate or analyse molecular dynamics trajectories. The library focuses on two key features: + + * **Memory efficiency.** + The size of trajectory data can quickly overwhelm the memory resources of your computer. + MDAnalysis typically accesses your trajectory by only loading data for one frame at a time. + This allows you to work with trajectories of any length without difficulty. + + * **Flexibility.** + MDAnalysis is constructed to be easily extensible. + If an analysis method is not already available in MDAnalysis, + you can write your own custom trajectory analysis with the building blocks provided. + If you need to read in a custom file format, you can construct your own Reader or Parser that will automatically get picked up when MDAnalysis is constructing a Universe from files. You can create and add your own labels for atoms, residues, or segments (called :ref:`topology attributes `) and relationships between atoms (e.g. bonds, angles). + + +Participating +============= + +MDAnalysis welcomes all contributions from its users. There are many ways you can help improve MDAnalysis, from asking questions on the `mdnalysis-discussion`_ mailing list, to raising issues on the `Issue Tracker`_, to adding your own code. Please see :ref:`contributing` for an introduction and guide to contributing to the code and documentation. + +.. important:: + + **Ground rules and expectations** + + The MDAnalysis community subscribes to a `Code of Conduct`_. By participating in this project and community, you agree to abide by its terms. Please read it. + + In general, we expect you to **be kind and thoughtful in your conversations around this project.** We all come from different backgrounds and projects, which means we will not always agree. Try to listen and understand why others hold their viewpoints in discussions. Rather than blaming each other, focus on helping to resolve issues and learning from mistakes. + + + +-------------- +Communications +-------------- + +Questions and discussions about MDAnalysis take place on the mailing lists and this repository’s `Issue Tracker`_. Anybody is welcome to join these conversations. Please ask questions about the usage of MDAnalysis on the `mdnalysis-discussion`_ mailing list, and report problems on the `Issue Tracker`_. + +Wherever possible, do not take these conversations to private channels, including contacting the maintainers directly. Keeping communication public means everybody can benefit and learn from the conversation. + +.. _`mdnalysis-discussion`: + https://groups.google.com/group/mdnalysis-discussion +.. _`Code of Conduct`: https://www.mdanalysis.org/pages/conduct/ +.. _`Issue Tracker`: https://github.com/MDAnalysis/mdanalysis/issues .. toctree:: :maxdepth: 1 :caption: Getting started + :hidden: installation examples/quickstart @@ -42,6 +77,7 @@ Gromacs_, CHARMM_, VMD_ and PyMol_ (see :ref:`selection-exporters`). It uses Num .. toctree:: :maxdepth: 1 :caption: Data structures + :hidden: universe atomgroup @@ -53,6 +89,7 @@ Gromacs_, CHARMM_, VMD_ and PyMol_ (see :ref:`selection-exporters`). It uses Num .. toctree:: :maxdepth: 1 :caption: Input/output + :hidden: reading_and_writing formats/index @@ -64,17 +101,28 @@ Gromacs_, CHARMM_, VMD_ and PyMol_ (see :ref:`selection-exporters`). It uses Num .. toctree:: :maxdepth: 1 :caption: Analysis + :hidden: examples/analysis/README examples/analysis/custom_trajectory_analysis .. toctree:: :maxdepth: 1 - :caption: Developer docs + :caption: Advanced + :hidden: standard_selections advanced_topology + datasets + +.. toctree:: + :maxdepth: 1 + :caption: Contributing + :hidden: + contributing + contributing_code + contributing_docs preparing_releases_and_hotfixes module_imports testing diff --git a/doc/source/installation.rst b/doc/source/installation.rst index 2a371e9e0..30caf1350 100644 --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -9,7 +9,7 @@ Currently, the conda releases only support serial calculations. If you plan to use the parallel OpenMP algorithms, you need to install MDAnalysis with pip and have a working OpenMP installation. -MDAnalysis has a separate *test suite* **MDAnalysisTests** that is required to run the test cases and examples. +MDAnalysis has a separate :ref:`test suite ` **MDAnalysisTests** that is required to run the test cases and examples. The test files change less frequently, take up around 90 MB of space, and are not needed for daily use of MDAnalysis. However, they are often used in examples, including many in this User Guide. If you are not interested in developing @@ -19,7 +19,7 @@ The tests are distributed separately from the main package. conda ===== -To install the latest stable version of MDAnalysis via Anaconda, use the following command: +To install the latest stable version of MDAnalysis via Anaconda, use the following command. This installs all dependencies needed for full analysis functionality (excluding external programs such as `HOLE`_: .. code-block:: bash @@ -39,17 +39,17 @@ To install the tests: pip ===== -To install the latest stable version of MDAnalysis via pip, use the following command: +The following command will install or upgrade the latest stable version of MDAnalysis via pip, with core dependencies. This means that some packages required by specific analysis modules will not be installed. .. code-block:: bash pip install --upgrade MDAnalysis -To upgrade: +If you need to install a fully-featured MDAnalysis, add the ``analysis`` tag. As with ``conda``, this will not install external programs such as `HOLE`_. .. code-block:: bash - pip install --upgrade MDAnalysis + pip install --upgrade MDAnalysis[analysis] To install/upgrade tests: @@ -87,3 +87,21 @@ The plugin `pytest-xdist `_ can be u pip install pytest-xdist pytest --disable-pytest-warnings --pyargs MDAnalysisTests --numprocesses 4 + + +Additional datasets +=================== + +:ref:`MDAnalysisData is an additional package ` with datasets that can be used in example tutorials. You can install it with ``conda`` or ``pip``: + +.. code-block:: bash + + # conda + conda install -c conda-forge mdanalysisdata + # pip + pip install --upgrade MDAnalysisData + +This installation does not download all the datasets; instead, the datasets are cached when they are first downloaded using a Python command. + + +.. _`HOLE`: http://www.holeprogram.org diff --git a/doc/source/testing.rst b/doc/source/testing.rst index c00a09287..3de9bdbb9 100644 --- a/doc/source/testing.rst +++ b/doc/source/testing.rst @@ -18,7 +18,7 @@ Whenever you add new code, you should create an appropriate test case that check #. More importantly, in the future we can always test that it is still working correctly. Unit tests are a crucial component of proper software engineering (see e.g. `Software Carpentry on Testing `_) and a large (and growing) test suite is one of the strengths of MDAnalysis. Adding tests is one of the most common requests after code is pushed to MDAnalysis. Therefore, -it is worth getting in the habit of writing tests ahead of time so this is never an issue. We strive for >85% our code to be covered by tests. +it is worth getting in the habit of writing tests ahead of time so this is never an issue. We strive for ≥90% our code to be covered by tests. We strongly encourage contributors to embrace `test-driven development `_. @@ -33,6 +33,150 @@ Like many packages, MDAnalysis uses `pytest `_ and some of the `numpy.testing `_ framework. +.. _run-test-suite: + +Running the test suite +====================== + +It is recommended that you run the tests from the ``mdanalysis/testsuite/MDAnalysisTests/`` directory. + +.. code-block:: bash + + cd testsuite/MDAnalysisTests + pytest --disable-pytest-warnings + +All tests should pass: no **FAIL** or **ERROR** cases should be triggered; *SKIPPED* or *XFAIL* are ok. For anything that fails or gives an error, ask on the `mdnalysis-discussion`_ mailing list or raise an issue on the `Issue Tracker `_. + +We use the ``--disable-pytest-warnings`` when the whole testsuite is running, as pytest raises a lot of false positives when we warn users about missing topology attributes. When running single tests or only single modules, consider running the tests *with* warnings enabled (i.e. without ``--disable-pytest-warnings``). This allows you to see if you trigger any un-caught deprecation warnings or other warnings in libraries we use. + +To run specific tests just specify the path to the test file: + +.. code-block:: bash + + pytest testsuite/MDAnalysisTests/analysis/test_align.py + + +Specific test classes inside test files, and even specific test methods, can also be specified: + +.. code-block:: bash + + # Test the entire TestContactMatrix class + pytest testsuite/MDAnalysisTests/analysis/test_analysis.py::TestContactMatrix + + # Test only test_sparse in the TestContactMatrix class + pytest testsuite/MDAnalysisTests/analysis/test_analysis.py::TestContactMatrix::test_sparse + + +This is very useful when you add a new test and want to check if it passes. However, before you push your code to GitHub, make sure that your test case runs and that *all other test cases are still passing*. + +------------------- +Testing in parallel +------------------- + +Running the tests serially can take some time, depending on the performance of your computer. You can speed this up by using the plugin `pytest-xdist `_ to run tests in parallel by specifying the ``--numprocesses`` option: + +.. code-block:: bash + + pip install pytest-xdist + pytest --disable-pytest-warnings --numprocesses 4 + + +You can try increasing the number of processes to speed up the test run. The number of processes you can use depends on your machine. + +------------- +Test coverage +------------- + +The tool `pytest-cov `_ can be used to generate the coverage report locally: + +.. code-block:: bash + + pip install pytest-cov + pytest --cov=MDAnalysis + + +Note: You can use the ``--numprocesses`` flag to run tests in parallel with the above command too. This will print the coverage statistic for every module in MDAnalysis at the end of a run. To get detailed line by +line statistics you can add the ``--cov-report=html`` flag. This will create a ``htmlcov`` folder (in the directory you run the command from) and there will be an ``index.html`` file in this folder. Open this file in your browser and you will be able to see overall statistics and detailed line coverage for each file. + +.. _continuous-integration: + +Continuous Integration tools +============================ + +When you submit your pull request, several continuous integration tools will run a suite of tests. These should all pass before your code can be merged into MDAnalysis. You can check tests locally by :ref:`running the test suite `. + +If your pull request fails immediately with an :ref:`appveyor` error, it is likely that you have merge conflicts with the latest code in the ``develop`` branch. :ref:`Rebase your code ` and update your branch by pushing your changes. + +If you get an error with :ref:`travis`, it is likely that you've failed a particular test. You should update your code and push again. + +If you get :ref:`codecov` errors, this means that your changes have not been adequately tested. Add new tests that address the "missed" lines, and push again. + +Ideally, you want all tests to pass. This will look like: + + .. image:: images/ci_checks_passed.png + +.. _appveyor: + +-------- +Appveyor +-------- + +`AppVeyor`_ is a continuous integration/continuous deployment service. MDAnalysis uses it for `testing builds on Windows`_. + +Builds are configured in the file ``.appveyor.yml``. If you add a new dependency to MDAnalysis, you will need to add it to the ``$CONDA_DEPENDENCIES`` or ``$PIP_DEPENDENCIES`` in ``.appveyor.yml`` to pass tests. + +.. _`testing builds on Windows`: https://ci.appveyor.com/project/orbeckst/mdanalysis + + +.. _travis: + +------ +Travis +------ + +`Travis is a continuous integration service `_ for Linux and MacOS. MDAnalysis uses it for exhaustive testing on Linux systems, and some testing on MacOS. If you add a new dependency to MDAnalysis, you will need to add it to the ``$CONDA_DEPENDENCIES`` or ``$PIP_DEPENDENCIES`` in ``.travis.yml`` to pass tests. + + +.. _codecov: + +------- +Codecov +------- + +Code coverage measures how many lines, and which lines, of code are executed by a test suite. Codecov is a service that turns code coverage reports into a single visual report. Each line is described as one of three categories: + + - a **hit** indicates that the source code was executed by the test suite. + - a **partial** indicates that the source code was not fully executed by the test suite; there are remaining branches that were not executed. + - a **miss** indicates that the source code was not executed by the test suite. + +Coverage is the ratio of ``hits / (sum of hit + partial + miss)``. See the `Codecov documentation `_ for more information. + +MDAnalysis aims for 90% code coverage; your pull request will fail the Codecov check if the coverage falls below 85%. You can increase coverage by writing futher tests. + +On your pull request, Codecov will leave a comment with three sections: + + - a visual map of the areas with coverage changes + + .. image:: images/codecov_report_map.png + + - a summary of changes in coverage + + .. image:: images/codecov_report_summary.png + + - a list of files with changes + + .. image:: images/codecov_report_files.png + +Clicking on one of those files will show the Codecov :guilabel:`Diff` view, highlighting the lines of code that have been missed by tests. In the image below, the column on the left hand side shows hits (green) and misses (red); the lighter colours highlighting the code show lines added (light green) or removed (light red). + + .. image:: images/codecov_diff.png + +Changing to the :guilabel:`Coverage Changes` view highlights how your additions have changed the test coverage. See the `documentation for viewing source code `_ for more information. + + .. image:: images/codecov_coverage_changes.png + +.. _write-new-tests: + Writing new tests ================= @@ -54,7 +198,7 @@ Use plain ``assert`` statements for comparing single values, e.g. :: def test_foo_is_length_3(foo): assert len(foo) == 3 -To check equality up to a certain precision, use ``assert_almost_equal`` from ``numpy.testing``. Do not manually round off the value and use plain ``assert`` statements. Do not use ``pytest.approx`` :: +To check equality up to a certain precision (e.g. floating point numbers and iterables of floats), use :func:`~numpy.testing.assert_almost_equal` from :mod:`numpy.testing`. Do not manually round off the value and use plain ``assert`` statements. Do not use ``pytest.approx``. :: from numpy.testing import assert_almost_equal @@ -63,14 +207,14 @@ To check equality up to a certain precision, use ``assert_almost_equal`` from `` u = mda.Universe(PDB_small) assert_almost_equal(u.atoms.positions, ref.atoms.positions) -To compare an iterable, use ``assert_equal`` from ``numpy.testing``. Do not iterate over and compare every single value. :: +To test for exact equality (e.g. integers, booleans, strings), use :func:`~numpy.testing.assert_equal` from :mod:`numpy.testing`. As with :func:`~numpy.testing.assert_almost_equal`, this should be used for iterables of exact values as well. Do not iterate over and compare every single value. :: from numpy.testing import assert_equal def test_equal_arrays(array1, array2): assert_equal(array1, array2) -Do not use ``assert_array_equal`` or ``assert_array_almost_equal`` from ``numpy.testing`` to compare array/array-like data structures. Instead, use ``assert_equal`` or ``assert_almost_equal``. The former set of functions equate arrays and scalars, while the latter do not: +Do not use ``assert_array_equal`` or ``assert_array_almost_equal`` from :mod:`numpy.testing` to compare array/array-like data structures. Instead, use :func:`~numpy.testing.assert_equal` or :func:`~numpy.testing.assert_almost_equal`. The former set of functions equate arrays and scalars, while the latter do not: .. ipython:: python :okexcept: @@ -79,12 +223,12 @@ Do not use ``assert_array_equal`` or ``assert_array_almost_equal`` from ``numpy. assert_array_equal([1], 1) assert_equal([1], 1) -Do not use anything from ``numpy.testing`` that depends on ``nose``, such as ``assert_raises``. +Do not use anything from :mod:`numpy.testing` that depends on ``nose``, such as ``assert_raises``. Testing exceptions and warnings ------------------------------- -Do not use ``assert_raises`` from ``numpy.testing`` or the ``pytest.mark.raises`` decorator to test for particular exceptions. Instead, use context managers:: +Do not use ``assert_raises`` from :mod:`numpy.testing` or the ``pytest.mark.raises`` decorator to test for particular exceptions. Instead, use context managers:: def test_for_error(): a = [1, 2, 3] @@ -98,13 +242,13 @@ Do not use ``assert_raises`` from ``numpy.testing`` or the ``pytest.mark.raises` Failing tests ------------- -To mark an expected failure, use ``pytest.mark.xfail`` decorator:: +To mark an expected failure, use :func:`pytest.mark.xfail` decorator:: @pytest.mark.xfail def tested_expected_failure(): assert 1 == 2 -To manually fail a test, make a call to ``pytest.fail``:: +To manually fail a test, make a call to :func:`pytest.fail`:: def test_open(self, tmpdir): outfile = str(tmpdir.join('lammps-writer-test.dcd')) @@ -117,7 +261,7 @@ To manually fail a test, make a call to ``pytest.fail``:: Skipping tests -------------- -To skip tests based on a condition, use ``pytest.mark.skipif(condition)`` decorator:: +To skip tests based on a condition, use :func:`pytest.mark.skipif(condition) ` decorator:: import numpy as np try: @@ -133,7 +277,7 @@ To skip tests based on a condition, use ``pytest.mark.skipif(condition)`` decora assert not np.shares_memory(original.ts.positions, copy.ts.positions) -To skip a test if a module is not available for importing, use ``pytest.importorskip('module_name')`` :: +To skip a test if a module is not available for importing, use :func:`pytest.importorskip('module_name') ` :: def test_write_trajectory_netCDF4(self, universe, outfile): pytest.importorskip("netCDF4") @@ -143,7 +287,7 @@ To skip a test if a module is not available for importing, use ``pytest.importor Fixtures -------- -Use `fixtures `_ as much as possible to reuse "resources" between test methods/functions. Pytest fixtures are functions that run before each test function that uses that fixture. A fixture is typically set up with the ``pytest.fixture`` decorator, over a function that returns the object you need:: +Use `fixtures `_ as much as possible to reuse "resources" between test methods/functions. Pytest fixtures are functions that run before each test function that uses that fixture. A fixture is typically set up with the :func:`pytest.fixture` decorator, over a function that returns the object you need:: @pytest.fixture def universe(self): @@ -165,7 +309,7 @@ The rule of thumb is to use the largest possible scope for the fixture to save t Testing the same function with different inputs ----------------------------------------------- -Use the ``pytest.mark.parametrize decorator`` to test the same function for different inputs rather than looping. These can be stacked:: +Use the :func:`pytest.mark.parametrize` decorator to test the same function for different inputs rather than looping. These can be stacked:: @pytest.mark.parametrize('pbc', (True, False)) @pytest.mark.parametrize('name, compound', (('molnums', 'molecules'), @@ -192,7 +336,7 @@ The code above runs ``test_center_of_mass_compounds_special`` 4 times with the f Temporary files and directories ------------------------------- -Do not use ``os.chdir()`` to change directories in tests, because it can break the tests in really weird ways (see `Issue 556`_). To use a temporary directory as the working directory, use the ``tmpdir.as_cwd`` context manager instead:: +Do not use :func:`os.chdir` to change directories in tests, because it can break the tests in really weird ways (see `Issue 556`_). To use a temporary directory as the working directory, use the :func:`tmpdir.as_cwd` context manager instead:: def test_write_no_args(self, u, tmpdir): # tmpdir is an in-built fixture with tmpdir.as_cwd(): @@ -234,150 +378,5 @@ If possible, re-use the existing data files in MDAnalysis for tests; this helps #. Add the files to the ``testsuite/MDAnalysisTests/data`` directory and appropriate file names and descriptions to ``testsuite/MDAnalysisTests/datafiles.py``. #. Make sure your new files are picked up by the pattern-matching in ``testsuite/setup.py`` (in the ``package_data`` dictionary). -Make sure that your test case runs and that *all other test cases are still passing*. - -.. _run-test-suite: - -Running the test suite -====================== - -It is recommended that you run the tests from the ``mdanalysis/testsuite/MDAnalysisTests/`` directory. - -.. code-block:: bash - - cd testsuite/MDAnalysisTests - pytest --disable-pytest-warnings - -All tests should pass: no **FAIL** or **ERROR** cases should be triggered; *SKIPPED* or *XFAIL* are ok. For anything that fails or gives an error, ask on the `mdnalysis-discussion`_ mailing list or raise an issue on the `Issue Tracker`_. - -We use the ``--disable-pytest-warnings`` when the whole testsuite is running, as pytest raises a lot of false positives when we warn users about missing topology attributes. When running single tests or only single modules, consider running the tests *with* warnings enabled (i.e. without ``--disable-pytest-warnings``). This allows you to see if you trigger any un-caught deprecation warnings or other warnings in libraries we use. - -To run specific tests just specify the path to the test file: - -.. code-block:: bash - - pytest testsuite/MDAnalysisTests/analysis/test_align.py - - -Specific test classes inside test files, and even specific test methods, can also be specified: - -.. code-block:: bash - - # Test the entire TestContactMatrix class - pytest testsuite/MDAnalysisTests/analysis/test_analysis.py::TestContactMatrix - - # Test only test_sparse in the TestContactMatrix class - pytest testsuite/MDAnalysisTests/analysis/test_analysis.py::TestContactMatrix::test_sparse - - -This is very useful when you add a new test and want to check if it passes. - -------------------- -Testing in parallel -------------------- - -Running the tests serially can take some time, depending on the performance of your computer. You can speed this up by using the plugin `pytest-xdist `_ to run tests in parallel by specifying the ``--numprocesses`` option: - -.. code-block:: bash - - pip install pytest-xdist - pytest --disable-pytest-warnings --numprocesses 4 - - -You can try increasing the number of processes to speed up the test run. The number of processes you can use depends on your machine. - -------------- -Test coverage -------------- - -The tool `pytest-cov `_ can be used to generate the coverage report locally: - -.. code-block:: bash - - pip install pytest-cov - pytest --cov=MDAnalysis - - -Note: You can use the ``--numprocesses`` flag to run tests in parallel with the above command too. This will print the coverage statistic for every module in MDAnalysis at the end of a run. To get detailed line by -line statistics you can add the ``--cov-report=html`` flag. This will create a ``htmlcov`` folder (in the directory you run the command from) and there will be an ``index.html`` file in this folder. Open this file in your browser and you will be able to see overall statistics and detailed line coverage for each file. - -.. _continuous-integration: - -Continuous Integration tools -============================ - -When you submit your pull request, several continuous integration tools will run a suite of tests. These should all pass before your code can be merged into MDAnalysis. You can check tests locally by :ref:`running the test suite `. - -If your pull request fails immediately with an :ref:`appveyor` error, it is likely that you have merge conflicts with the latest code in the ``develop`` branch. :ref:`Rebase your code ` and update your branch by pushing your changes. - -If you get an error with :ref:`travis`, it is likely that you've failed a particular test. You should update your code and push again. - -If you get :ref:`codecov` errors, this means that your changes have not been adequately tested. Add new tests that address the "missed" lines, and push again. - -Ideally, you want all tests to pass. This will look like: - - .. image:: images/ci_checks_passed.png - -.. _appveyor: - --------- -Appveyor --------- - -`AppVeyor`_ is a continuous integration/continuous deployment service. MDAnalysis uses it for `testing builds on Windows`_. - -Builds are configured in the file ``.appveyor.yml``. If you add a new dependency to MDAnalysis, you will need to add it to the ``$CONDA_DEPENDENCIES`` or ``$PIP_DEPENDENCIES`` in ``.appveyor.yml`` to pass tests. - -.. _`testing builds on Windows`: https://ci.appveyor.com/project/orbeckst/mdanalysis - - -.. _travis: - ------- -Travis ------- - -`Travis is a continuous integration service `_ for Linux and MacOS. MDAnalysis uses it for exhaustive testing on Linux systems, and some testing on MacOS. If you add a new dependency to MDAnalysis, you will need to add it to the ``$CONDA_DEPENDENCIES`` or ``$PIP_DEPENDENCIES`` in ``.travis.yml`` to pass tests. - - -.. _codecov: - -------- -Codecov -------- - -Code coverage measures how many lines, and which lines, of code are executed by a test suite. Codecov is a service that turns code coverage reports into a single visual report. Each line is described as one of three categories: - - - a **hit** indicates that the source code was executed by the test suite. - - a **partial** indicates that the source code was not fully executed by the test suite; there are remaining branches that were not executed. - - a **miss** indicates that the source code was not executed by the test suite. - -Coverage is the ratio of ``hits / (sum of hit + partial + miss)``. See the `Codecov documentation `_ for more information. - -MDAnalysis aims for 90% code coverage; your pull request will fail the Codecov check if the coverage falls below 85%. You can increase coverage by writing futher tests. - -On your pull request, Codecov will leave a comment with three sections: - - - a visual map of the areas with coverage changes - - .. image:: images/codecov_report_map.png - - - a summary of changes in coverage - - .. image:: images/codecov_report_summary.png - - - a list of files with changes - - .. image:: images/codecov_report_files.png - -Clicking on one of those files will show the Codecov :guilabel:`Diff` view, highlighting the lines of code that have been missed by tests. In the image below, the column on the left hand side shows hits (green) and misses (red); the lighter colours highlighting the code show lines added (light green) or removed (light red). - - .. image:: images/codecov_diff.png - -Changing to the :guilabel:`Coverage Changes` view highlights how your additions have changed the test coverage. See the `documentation for viewing source code `_ for more information. - - .. image:: images/codecov_coverage_changes.png - -.. _Issue Tracker: https://github.com/MDAnalysis/mdanalysis/issues .. _`mdnalysis-discussion`: http://groups.google.com/group/mdnalysis-discussion \ No newline at end of file