Skip to content

Commit

Permalink
Revamp prterun show_help files to integrate with Sphinx
Browse files Browse the repository at this point in the history
This commit represents a major reorganization and revamp of prterun's
show_help files.  The overriding purpose is to allow downstream
projects -- such as MPI	implementations	-- to include PRRTE's Sphinx
(.rst) documentation in their own.

This entailed the following:

* The existing "show help" .txt files are used by the existing "<cmd>
  --help <topic>" CLI system.  It was important to preserve this CLI
  capability.

* However, we wanted this same content to also appear in various man
  pages (and therefore also HTML documentation).  Hence, we moved the
  existing "show_help" .txt files to a new location: src/docs/
  (vs. docs/, where the main PRRTE documentation lives).

  * After various experiments, we determined that we just could not
    have both the main PRRTE Sphinx docs and this new class of
    show_help/man-page-content combination docs in the same Sphinx
    docs tree.  The structure and layout of how we need to render
    these two sets of docs are quite different.

    Instead, as described below, we took a different approach: have
    "building blocks" of content that can be assembled in different
    ways that are ultimately rendered with the structure and layout
    required for each environment.

* src/docs/ contains 2 directories:

  * prrte-rst-content: this directory contains a README.txt file that
    both PRRTE docs authors and downstream consumers of the RST
    content should read.  It explains how both how to author/maintain
    the docs in that directory, as well as how to integrate the
    contents of this directory into downstream project RST docroots,
    and how to include the content.

    Essentially, this directory is the main source of prterun CLI
    documentation content.  Each topic is split into its own file (see
    the README for more info) so that it can be RST-included in other
    .rst files as relevant.  Think of these files as individual
    building blocks of content that can be assembled in multiple
    different ways.

    The text content in this directory is mostly content from previous
    show_help-style text files moved into this directory, split into
    individual .rst files, and then converted to RST syntax.  Some of
    the tables ended up being nicer / easier to maintain, for example.

  * show-help-files: the contents of this directory are rendered to
    what ultimately become the "show_help" text files.

    This directory effectively RST-includes the files from the
    prrte-rst-content directory into files and renders them as
    INI-style "show_help" text files (i.e., suitable for reading by
    the prte_show_help() subsystem).

* The src/docs/prrte-rst-content is also RST-included in the top-level
  docs/ .rst files (va a sym link).  That allows the same "building
  blocks" of documentation content to be included in both the
  show_help files and the HTML/man pages.

* Note that in distribution tarballs, pre-built HTML docs, norff man
  pages, and INI-style "show_help" text files will be included in the
  tarball -- end users do not need to have Sphinx available.

  For git clones, where none of the Sphinx-generated content is
  available (i.e., HTML docs, nroff man pages, show_help text files),
  Sphinx is also still not required (but it *is* still required to
  "make dist").  If Sphinx is not available when "configure" is
  invoked, dummy INI-style "show_help" text files are generated (that
  basically say "You don't have Sphinx, so you don't get help here").

That is the overall scheme.  There were many other minor points
included in this work, including (but not limited to):

* Moving existing explanations of hosts and hostfiles -- and expanding
  on them -- to the main docs.

* Moving explanations of relative indexting to the main docs.

* Moving explanations of notifications to the main docs.

* Moving listing of deprecated CLI options to the main docs.

* Moving explanations of diagnostics to the main docs.

* Moving most process placement explanations and examples to the main
  docs.

* Moving explanations of the session directory to the main docs.

* Adding psched.1 and pterm.1 man pages.  More work needs to be done
  here.

* Slightly improve prte.1, prte_info.1, prted.1, prterun.1, and prun.1
  man pages.  More work needs to be done here.

It is expected that the text content of all the docs will continue to
be improved over time.  The focus of this commit was not to make the
content perfect, but rather to create a first-gen infrastructure for
having a single source of documentation content that can be rendered
by Sphinx in 3 ways (HTML docs, nroff man pages, and show_help files),
and also included in downstream RST documentation.

Signed-off-by: Ralph Castain <rhc@pmix.org>
Signed-off-by: Jeff Squyres <jeff@squyres.com>
  • Loading branch information
rhc54 authored and jsquyres committed Sep 10, 2023
1 parent fe61ade commit 8b66091
Show file tree
Hide file tree
Showing 106 changed files with 5,919 additions and 3,999 deletions.
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,6 @@ vc70.pdb
.hgignore_local
stamp-h?
AUTHORS
docs-venv/

ar-lib
ylwrap
Expand Down Expand Up @@ -190,7 +189,12 @@ docs/_build
docs/_static
docs/_static/css/custom.css
docs/_templates
src/docs/_build
src/docs/mca/mca.rst
src/docs/mca/help*rst
__pycache__

# Common Python virtual environment directory names
venv
py??
docs-venv
8 changes: 8 additions & 0 deletions Makefile.prte-rules
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,11 @@ prte__v_SPHINX_HTML_0 = @echo " GENERATE HTML docs";
PRTE_V_SPHINX_MAN = $(prte__v_SPHINX_MAN_$V)
prte__v_SPHINX_MAN_ = $(prte__v_SPHINX_MAN_$AM_DEFAULT_VERBOSITY)
prte__v_SPHINX_MAN_0 = @echo " GENERATE man pages";

PRTE_V_TXT = $(prte__v_SPHINX_TXT_$V)
prte__v_TXT_ = $(prte__v_SPHINX_TXT_$AM_DEFAULT_VERBOSITY)
prte__v_TXT_0 = @echo " GENERATE text files";

PRTE_V_LN_S = $(prte__v_LN_S_$V)
prte__v_LN_S_ = $(prte__v_LN_S_$AM_DEFAULT_VERBOSITY)
prte__v_LN_S_0 = @echo " LN_S " `basename $@`;
10 changes: 9 additions & 1 deletion autogen.pl
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@
# Copyright (c) 2015 Research Organization for Information Science
# and Technology (RIST). All rights reserved.
# Copyright (c) 2015 IBM Corporation. All rights reserved.
#
# Copyright (c) 2021-2023 Nanook Consulting. All rights reserved.
# Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
#
# $COPYRIGHT$
#
# Additional copyrights may follow
Expand Down Expand Up @@ -146,6 +147,13 @@ sub mca_process_component {
verbose " Found configure.m4 file\n";
}

# Does this directory have a
# help-{framework}-{component}.rst file?
if (-f "$cdir/help-$framework-$component.rst") {
$found_component->{"rst"} = 1;
verbose " Found help-$framework-$component.rst file\n";
}

# Push the results onto the $mca_found hash array
push(@{$mca_found->{$framework}->{"components"}},
$found_component);
Expand Down
14 changes: 13 additions & 1 deletion configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
# All Rights reserved.
# Copyright (c) 2021-2023 Nanook Consulting. All rights reserved.
# Copyright (c) 2021 FUJITSU LIMITED. All rights reserved.
# Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights may follow
Expand Down Expand Up @@ -57,6 +58,14 @@ AC_CONFIG_MACRO_DIR(config)

OAC_PUSH_PREFIX([PRTE])

# Get the absolute version of the srcdir. We don't use "readlink -f",
# because that unfortunately isn't portable (cough cough macOS cough
# cough).
save=$(pwd)
cd $srcdir
abs_srcdir=$(pwd)
cd $save

# autotools expects to perform tests without interference
# from user-provided CFLAGS, particularly -Werror flags.
# Search for them here and cache any we find
Expand Down Expand Up @@ -705,7 +714,7 @@ AC_INCLUDES_DEFAULT
#endif])

#
# Setup HTML and man page processing
# Setup Sphinx processing
#

OAC_SETUP_SPHINX([$srcdir/docs/_build/html/index.html], [])
Expand Down Expand Up @@ -991,6 +1000,9 @@ AC_CONFIG_FILES([
include/Makefile
include/prte_version.h
docs/Makefile
src/docs/Makefile
src/docs/show-help-files/Makefile
src/docs/prrte-rst-content/Makefile
])

PRTE_CONFIG_FILES
Expand Down
19 changes: 12 additions & 7 deletions docs/Makefile.am
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#
# Copyright (c) 2022-2023 Cisco Systems, Inc. All rights reserved.
# Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
#
# Copyright (c) 2023 Nanook Consulting. All rights reserved.
# $COPYRIGHT$
Expand Down Expand Up @@ -33,20 +34,20 @@ SPHINX_OPTS ?= -W --keep-going -j auto
# Note: it is significantly more convenient to list all the source
# files here using wildcards (vs. listing every single .rst file).
# However, it is necessary to list $(srcdir) when using wildcards.
TEXT_SOURCE_FILES = # if you have any .txt files, list them here
IMAGE_SOURCE_FILES = # if you have any, list them here
RST_SOURCE_FILES = \
RST_SOURCE_FILES = \
$(srcdir)/*.rst \
$(srcdir)/prrte-rst-content/*.rst \
$(srcdir)/placement/*.rst \
$(srcdir)/hosts/*.rst \
$(srcdir)/developers/*.rst \
$(srcdir)/man/*.rst \
$(srcdir)/man/man1/*.rst \
$(srcdir)/man/man5/*.rst

EXTRA_DIST = \
EXTRA_DIST = \
requirements.txt \
_templates/configurator.html \
$(SPHINX_CONFIG) \
$(TEXT_SOURCE_FILES) \
$(IMAGE_SOURCE_FILES) \
$(RST_SOURCE_FILES)

###########################################################################
Expand All @@ -63,7 +64,9 @@ PRTE_MAN1 = \
prte_info.1 \
prted.1 \
prterun.1 \
prun.1
prun.1 \
psched.1 \
pterm.1

PRTE_MAN5 = \
prte.5
Expand All @@ -76,6 +79,8 @@ PRTE_MAN1_BUILT = $(PRTE_MAN1:%.1=$(MAN_OUTDIR)/%.1)
PRTE_MAN5_RST = $(PRTE_MAN5:%.5=man/man5/%.5.rst)
PRTE_MAN5_BUILT = $(PRTE_MAN5:%.5=$(MAN_OUTDIR)/%.5)

#-------------------

EXTRA_DIST += \
$(PRTE_MAN1_BUILT) \
$(PRTE_MAN5_BUILT)
Expand Down
22 changes: 17 additions & 5 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,13 @@
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
import os
import sys

# -- Project information -----------------------------------------------------

needs_sphinx = '4.2'

import datetime
year = datetime.datetime.now().year

Expand All @@ -25,8 +26,15 @@

# The full version, including alpha/beta/rc tags
# Read the PRRTE version from the VERSION file
with open("../VERSION") as fp:
prte_lines = fp.readlines()
# Note: this conf file lives in 2 different directories, so find the
# VERSION file relative to where we're running right now.
prte_lines = None
for dir in ['..', '../../..']:
file = f'{dir}/VERSION'
if os.path.exists(file):
with open(file) as fp:
prte_lines = fp.readlines()
break

prte_data = dict()
for prte_line in prte_lines:
Expand Down Expand Up @@ -171,6 +179,10 @@
tuple = (full_filename_without_rst, base_name, '', '', section)
man_pages.append(tuple)

# -- Options for TEXT output ------------------------------------------------

text_sectionchars = '=-$#@!`'

# -- PRRTE-specific options -----------------------------------------------

# This prolog is included in every file. Put common stuff here.
Expand Down
36 changes: 36 additions & 0 deletions docs/hosts/cli.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
.. _hosts-cli-label:

Listing Hosts on the Command Line
=================================

Many PRRTE commands accept the ``--host`` CLI parameter.
``--host`` accepts a comma-delimited list of tokens of the form:

.. code::
host[:slots]
The ``host`` token can be either:

* A name that resolves to an IP address, or
* An IP address

.. note:: The names and/or IP addresses of hosts are *only* used for
identifying the target host on which to launch. They are
*not* used for determining which network interfaces are used
by applications (e.g., MPI or other network-based
applications).

For network-based applications, consult their documentation
for how to specify which network interfaces are used.

The optional integer ``:slots`` parameter tells PRRTE the maximum
number of slots to use on that host (:ref:`see this section
<placement-definition-of-slot-label>` for a description of what a
"slot" is).

For example:

.. code::
prterun --host node1:10,node2,node3:5 ...
41 changes: 41 additions & 0 deletions docs/hosts/hostfiles.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
Hostfiles
=========

Hostfiles (sometimes called "machine files") are a combination of two
things:

#. A listing of hosts on which to launch processes.
#. Optionally, limit the number of processes which can be launched on
each host.

Syntax
------

Hostfile syntax consists of one node name on each line, optionally
including a designated number of "slots":

.. code:: sh
# This is a comment line, and will be ignored
node01 slots=10
node13 slots=5
node15
node16
node17 slots=3
...
Blank lines and lines beginning with a ``#`` are ignored.

A "slot" is the PRRTE term for an allocatable unit where we can launch
a process. :ref:`See this section
<placement-definition-of-slot-label>` for a longer description of
slots.

In the absence of the ``slot`` parameter, PRRTE will assign either the
number of slots to be the number of CPUs detected on the node or the
resource manager-assigned value if operating in the presence of an
RM.

.. important:: If using a resource manager, the user-specified number
of slots is capped by the RM-assigned value.
19 changes: 19 additions & 0 deletions docs/hosts/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
Host specification
==================

PRRTE identifies hosts on which to launch processes by:

#. A Resource Manager (RM) telling PRRTE which hosts to use
#. The user providing a hostfile (also sometimes called a "machine
file")
#. The user providing a list of hosts on the command line

The sections below describe each of these in more detail.

.. toctree::
:maxdepth: 1

rm
hostfiles
cli
relative-indexing
Loading

0 comments on commit 8b66091

Please sign in to comment.