Merge pull request #258 from ilaflott/update-fre-cmor-doc3

Update fre cmor doc

.github/workflows/create_test_conda_env.yml

@@ -27,9 +27,9 @@ jobs:
           # add conda env's executables to github's PATH equiv.
           echo $CONDA/envs/fre-cli/bin >> $GITHUB_PATH
 
-          ## use *conda environment's pip* to install fre-cli
-          ## called w/ full path to conda's python for explicitness
-          ## called as a module (-m pip) for explicitness
+          # use *conda environment's pip* to install fre-cli
+          # called w/ full path to conda's python for explicitness
+          # called as a module (-m pip) for explicitness
           $CONDA/envs/fre-cli/bin/python -m pip install --prefix $CONDA/envs/fre-cli .
 
       - name: Run pytest in fre-cli environment

docs/contributing_to_doc.rst

@@ -12,35 +12,51 @@ How to Contribute to ``fre-cli``'s documentation
 
 
 
-1. fork and poke at the settings
-
-   * Fork ``fre-cli`` on github
-   * On github, navigate to your ``fre-cli`` fork, and click “settings”
-   * In “settings”, click “pages”
-   * In “pages”, under “build and deployment”, make sure “source” is set to “Deploy from a branch”
-   * Under that, find “Branch”, make sure the branch selected is ``gh-pages``
-   * The branch ``gh-pages`` is "automagic”- i.e. do not change anything about it nor create a new one,
-	 nor interact with anything in that branch directly
-
-
-2. enable workflows for your fork
-
-   * note: this step may depend on user-specific settings!
-   * Back on top where you found “settings”, find and click “actions” to the left
-   * Enable running the workflow actions assoc with the ``fre-cli`` repo under ``.github/workflows``
-
-
-3. run your fork's first workflow
-
-   * The documentation builds as the last steps to ``create_test_conda_env.yml`` when theres a push to ``main``
-   * To get your first workflow run on your fork, comment out the ``github.ref == ‘refs/heads/main’`` bit
-	 so that it runs when you push to any branch, and make a small, trivial, commit somewhere to your
-	 remote fork
-   * You should be able to find the deployed webpage from a successful workflow at
-	 https://your_username.github.io/fre-cli (if you did not change the fork’s name from ``fre-cli``, that is)
-   * If you’re only editing docs, you can make the turn-around time on your workflow ~3 min faster by
-	 commenting-out the ``pylint`` and ``pytest`` steps in ``create_test_conda_env.yml``, and disable running the
-	 ``build_conda.yml`` workflow
-
-
-
+fork and poke at the settings
+-----------------------------
+
+* Fork ``fre-cli`` on github
+
+* On github, navigate to your ``fre-cli`` fork, and click “settings”
+
+* In “settings”, click “pages”
+
+* In “pages”, under “build and deployment”, make sure “source” is set to “Deploy from a branch”
+
+* Under that, find “Branch”, make sure the branch selected is ``gh-pages``
+
+* The branch ``gh-pages`` is "automagic”- i.e. do not change anything about it nor create a new one,
+  nor interact with anything in that branch directly
+
+
+enable workflows for your fork
+------------------------------
+
+note: this step may depend on user-specific settings!
+
+* Back on top where you found “settings”, find and click “actions” to the left
+
+* Enable running the workflow actions assoc with the ``fre-cli`` repo under ``.github/workflows``
+
+
+run your fork's first workflow
+------------------------------
+
+* The documentation builds as the last steps to ``create_test_conda_env.yml`` when theres a push to ``main``
+
+* To get your first workflow run on your fork, comment out the ``github.ref == ‘refs/heads/main’`` bit
+  so that it runs when you push to any branch, and make a small, trivial, commit somewhere to your
+  remote fork
+
+* You should be able to find the deployed webpage from a successful workflow at
+  https://your_username.github.io/fre-cli (if you did not change the fork’s name from ``fre-cli``, that is)
+
+* If you’re only editing docs, you can make the turn-around time on your workflow ~3 min faster by
+  commenting-out the ``pylint`` and ``pytest`` steps in ``create_test_conda_env.yml``, and disable running the
+  ``build_conda.yml`` workflow
+
+
+
+Other Helpful Things
+====================
+`restructured text cheat-sheet <https://gist.github.com/SMotaal/24006b13b354e6edad0c486749171a70#sections>`_

docs/fre_cmor.rst

@@ -1,5 +1,91 @@
 .. last updated Nov 2024
+
 ``run``
--------   
+-------
+
+``fre cmor run`` rewrites climate model output files in a target directory in a CMIP-compliant manner
+for downstream publishing. It accepts 6 arguments, only one being optional. A brief description of each:
+
+
+arguments
+~~~~~~~~~
+
+* (required) ``-d, --indir TEXT``, input directory containing netCDF files to CMORize.
+
+  - all netCDF files within ``indir`` will have their filename checked for local variables
+    specified in ``varlist`` as keys, and ISO datetime strings extracted and kept in a list
+    for later iteration over target files
+
+  - a debugging-oriented boolean flag constant at the top of ``cmor_mixer.py``, if ``True``
+    will process one file of all files found within ``indir`` and cease processing for that
+    variable after succeeding on one file
+
+* (required) ``-l, --varlist TEXT``, path to variable list dictionary.
+
+  - each entry in the variable list dictionary corresponds to a key/value pair
+
+  - the key (local variable) is used for ID'ing files within ``indir`` to be processed
+
+  - associated with the key (local variable), is the value (target variable), which should
+    be the label attached to the data within the targeted file(s)
+
+* (required) ``-r, --table_config TEXT``, path to MIP json configuration holding variable
+  metadata.
+
+  - typically, this is to be provided by a data-request associated with the MIP and
+    participating experiments
+
+* (required) ``-p, --exp_config TEXT``, path to json configuration holding experiment/model
+  metadata
+
+  - contains e.g. ``grid_label``, and points to other important configuration files
+    associated with the MIP
+
+  - the other configuration files are typically housing metadata associated with ``coordinates``,
+    ``formula_terms``, and controlled-vocabulary (``CV``).
+
+* (required) ``-o, --outdir TEXT``, path-prefix inwhich the output directory structure is created.
+
+  - further output-directories and structure/template information is specified specified in ``exp_config``
+
+  - in addition to the output-structure/template used, an additional directory corresponding to the
+    date the CMORizing was done is created near the end of the directory tree structure
+
+* (optional) ``-v, --opt_var_name TEXT``, a specific variable to target for processing
+
+  - largely a debugging convenience functionality, this can be helpful for targeting more specific
+    input files as desired. 
+
+
+examples
+~~~~~~~~
+with a local clone of ``fre-cli``, the following call should work out-of-the-box from
+the root directory of the repository.
+
+.. code-block:: python
+
+                fre cmor run \
+                   -d fre/tests/test_files/ocean_sos_var_file \
+                   -l fre/tests/test_files/varlist \
+                   -r fre/tests/test_files/cmip6-cmor-tables/Tables/CMIP6_Omon.json \
+                   -p fre/tests/test_files/CMOR_input_example.json \
+                   -o fre/tests/test_files/outdir
+
+background
+~~~~~~~~~~
+
+The bulk of this routine is housed in ``fre/cmor/cmor_mixer.py``, which is a rewritten version of
+Sergey Malyshev's original ``CMORcommander.py`` script, utilized during GFDL's CMIP6 publishing run.
+
+This code is dependent on two primary json configuration files- a MIP
+variable table and another containing experiment (i.e. model) specific metdata (e.g. grid) to append
+to the output netCDF file headers, in addition to other configuration options such as output directory
+name specification, output path templates, and specification of other json configuration files containing
+controlled-vocabulary (CV), coordinate, and formula term conventions for rewriting the output metadata.
+
+
+
+
+
 
 

docs/index.rst

@@ -2,9 +2,9 @@
    You can adapt this file completely to your liking, but it should at least contain the root
    \`toctree\` directive (no backslashes)
 
-===================================
-Welcome to Fre-Cli's documentation!
-===================================
+=======================================
+Welcome to ``fre-cli``'s documentation!
+=======================================
 
 .. the entry in the toc must be the .rst filename.
    what shows in the webpage is the first header or title

docs/setup.rst

@@ -30,6 +30,6 @@ activate the empty environment:
 append necessary channels:
 ``conda config --append channels noaa-gfdl; conda config --append channels conda-forge;``
 
-install ``fre-cli`` into the activated environment from `the GFDL conda channel https://anaconda.org/NOAA-GFDL/fre-cli`_ :
+install ``fre-cli`` into the activated environment from `the GFDL conda channel <https://anaconda.org/NOAA-GFDL/fre-cli>`_ :
 ``conda install noaa-gfdl::fre-cli``
 

docs/usage.rst

@@ -39,6 +39,14 @@ Brief rundown of commands also provided below:
 ``fre cmor``
 ============
 
+* See also, ``fre cmor``'s `README <https://github.com/NOAA-GFDL/fre-cli/blob/main/fre/cmor/README.md>`_
+* See also, ``fre cmor``'s `project board <https://github.com/orgs/NOAA-GFDL/projects/35>`_
+
+This set of tools leverages the external ``cmor`` python package within the ``fre`` ecosystem. ``cmor`` is an
+acronym for "climate model output rewriter". The process of rewriting model-specific output files for model
+intercomparisons (MIPs) using the ``cmor`` module is, quite cleverly, referred to as "CMORizing".
+
+
 .. include:: fre_cmor.rst