Merge pull request #4 from OZGrav/new_models

Documentation upgrade

.github/workflows/pytest.yaml

@@ -0,0 +1,26 @@
+name: Pytest
+
+on: [push]
+
+jobs:
+    build:
+
+        runs-on: ubuntu-latest
+        strategy:
+          matrix:
+            python-version: ["3.10"]
+
+        steps:
+            - uses: actions/checkout@v3
+            - name: Set up Python ${{ matrix.python-version }}
+              uses: actions/setup-python@v4
+              with:
+                python-version: ${{ matrix.python-version }}
+            - name: Install dependencies
+              run: |
+                python -m pip install --upgrade pip
+                pip install pytest
+                pip install .
+            - name: Test with pytest
+              run: |
+                pytest

.readthedocs.yaml

@@ -10,12 +10,18 @@ build:
   os: ubuntu-22.04
   tools:
     python: "3.11"
+  jobs:
+    post_create_environment:
+      # Install poetry
+      # https://python-poetry.org/docs/#installing-manually
+      - pip install poetry
+      # Tell poetry to not use a virtual environment
+      - poetry config virtualenvs.create false
+    post_install:
+      # Install dependencies with 'docs' dependency group
+      # https://python-poetry.org/docs/managing-dependencies/#dependency-groups
+      - poetry install --with docs
 
 # Build documentation in the "docs/" directory with Sphinx
 sphinx:
   configuration: docs/conf.py
-
-# Optionally declare the Python requirements required to build your docs
-python:
-  install:
-    - requirements: docs/requirements.txt

README.md

@@ -1,4 +1,8 @@
 # meerdb
+
+[![Documentation Status](https://readthedocs.org/projects/psrdb/badge/?version=latest)](https://psrdb.readthedocs.io/en/latest/?badge=latest)
+![example workflow](https://github.com/OZGrav/psrdb/actions/workflows/pytest.yaml/badge.svg)
+
 Command line interface and python package for interacting with the MeerTime data portal
 
 ## Installation

docs/cli.rst

@@ -0,0 +1,122 @@
+Command Line Interface
+======================
+
+
+
+
+Pulsar
+------
+
+.. argparse::
+    :module: psrdb.tables.pulsar
+    :func: get_parsers
+    :prog: psrdb pulsar
+    :nodefault:
+
+Telescope
+---------
+
+.. argparse::
+    :module: psrdb.tables.telescope
+    :func: get_parsers
+    :prog: psrdb telescope
+    :nodefault:
+
+MainProject
+-----------
+
+.. argparse::
+    :module: psrdb.tables.main_project
+    :func: get_parsers
+    :prog: psrdb main_project
+    :nodefault:
+
+Project
+-------
+
+.. argparse::
+    :module: psrdb.tables.project
+    :func: get_parsers
+    :prog: psrdb project
+    :nodefault:
+
+Ephemeris
+---------
+
+.. argparse::
+    :module: psrdb.tables.ephemeris
+    :func: get_parsers
+    :prog: psrdb  ephemeris
+    :nodefault:
+
+Template
+--------
+
+.. argparse::
+    :module: psrdb.tables.template
+    :func: get_parsers
+    :prog: psrdb template
+    :nodefault:
+
+Calibration
+------------
+
+.. argparse::
+    :module: psrdb.tables.calibration
+    :func: get_parsers
+    :prog: psrdb calibration
+    :nodefault:
+
+Observation
+-----------
+
+.. argparse::
+    :module: psrdb.tables.observation
+    :func: get_parsers
+    :prog: psrdb observation
+    :nodefault:
+
+PipelineRun
+-----------
+
+.. argparse::
+    :module: psrdb.tables.pipeline_run
+    :func: get_parsers
+    :prog: psrdb pipeline_run
+    :nodefault:
+
+PulsarFoldResult
+----------------
+
+.. argparse::
+    :module: psrdb.tables.pulsar_fold_result
+    :func: get_parsers
+    :prog: psrdb pulsar_fold_result
+    :nodefault:
+
+PipelineImage
+-------------
+
+.. argparse::
+    :module: psrdb.tables.pipeline_image
+    :func: get_parsers
+    :prog: psrdb pipeline_image
+    :nodefault:
+
+ToA
+---
+
+.. argparse::
+    :module: psrdb.tables.toa
+    :func: get_parsers
+    :prog: psrdb toa
+    :nodefault:
+
+Residual
+--------
+
+.. argparse::
+    :module: psrdb.tables.residual
+    :func: get_parsers
+    :prog: psrdb residual
+    :nodefault:

docs/conf.py

@@ -29,6 +29,7 @@
     'sphinx.ext.githubpages',
     'numpydoc',
     'myst_parser',
+    'sphinxarg.ext',
 ]
 
 templates_path = ['_templates']
@@ -50,4 +51,6 @@
 html_js_files = [
     'https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js',
     '_static/flyout.js'
-]
+]
+
+numpydoc_show_class_members = False

docs/how_to_use.md

@@ -1,11 +1,85 @@
 # How to use
 
-The command line interface can be used to list various tables in the database.
-The first arguments is the table name and the second is the action.
+The command line interface can be used to interact with the various database objects (also known as tables).
+The first arguments is the object name and the second is the action.
 For example if I wanted to list all observations of pulsar J1652-4838 I could use:
 
 ```
-psrdb observations list --pulsar J1652-4838
+psrdb observation list --pulsar J1652-4838
 ```
 
-If you're unsure which arguments you require you can use `-h` to list your options.
+If you're unsure which arguments you require you can use `-h` to list your options
+or look at the [Command Line Interface](cli) documentation page.
+
+Most database objects only have a `list` command but others (such as `toa` and `pulsar_fold_result`) also have a `download` command
+which will download data from the tables to a csv.
+
+
+## Pulsar Fold Result Download Example
+
+For example the `pulsar_fold_result` table contains the processing results such as the flux density,
+the dispersion measure, the rotation measure signal to noise of each observation.
+You can download the results for a given pulsar using:
+
+```
+psrdb pulsar_fold_result download J1652-4838
+```
+
+Which will download the results to a csv file called `pulsar_fold_result_J1652-4838.csv` in the current directory that contains data that looks like:
+
+```
+ID,UTC Start,Observing band,Duration (s),DM (pc cm^-3),DM error (pc cm^-3),DM epoch (MJD),DM chi2r,DM tres,SN,Flux (mJy),RM (rad m^-2),RM error (rad m^-2),RFI zapped (%)
+UHVsc2FyRm9sZFJlc3VsdE5vZGU6Mjkx,2023-06-16T17:12:10+00:00,LBAND,863.54886459813,188.17506533670348,0.0030674779606059722,59000.0,5.9886,5.9886,190.378768920898,1.151,-29.451,0.63,0.09469918224299065
+UHVsc2FyRm9sZFJlc3VsdE5vZGU6NDI0,2021-01-19T06:13:30+00:00,LBAND,1135.5898640747655,188.16665101512618,0.0033251138173515647,59000.0,7.2091,7.2091,226.744201660156,1.131,-33.6916,0.15,0.10732491134751773
+UHVsc2FyRm9sZFJlc3VsdE5vZGU6OTY0,2020-02-02T04:50:36+00:00,LBAND,1242.5246983177572,188.1714516981729,0.0030512582494712425,59000.0,5.6147,5.6147,224.331237792969,1.139,-42.526,0.34,0.06804886923421406
+...
+```
+
+[Pandas](https://pandas.pydata.org/docs/) is an excellent tool for filtering and analysising this data.
+For example, say you wanted to plot the flux density against the date for all bring observations in LBAND you could do the following:
+
+```
+import pandas as pd
+import matplotlib.pyplot as plt
+
+# Read your data into a Pandas DataFrame
+df = pd.read_csv('pulsar_fold_result_J1652-4838.csv')
+
+# Filter the data to only include LBAND observations
+df = df[df['Observing band'] == 'LBAND']
+# Filter the data to only include bright observations
+df = df[df['SN'] > 20]
+
+
+# Create a line plot
+x = pd.to_datetime(df['UTC Start'])
+y = df['Flux (mJy)']
+plt.scatter(x, y)
+
+# Add labels and a title
+plt.xlabel('Date (UTC)')
+plt.ylabel('Flux (mJy)')
+plt.gcf().autofmt_xdate()
+plt.savefig('J1652-4838_flux.png')
+```
+
+Which would make a plot like this:
+
+![J1652-4838_flux.png](figures/J1652-4838_flux.png)
+
+Which is not quite ready for publication but is a good starting point.
+
+
+## ToA Download Example
+
+The `toa` table contains the times of arrival for each observation.
+The ToAs can be generated a number of ways so to make sure you only download the ToAs you require,
+always use the `--nchan` option and either the `--minimum_nsubs` or `--maximum_nsubs` option.
+
+For example if I only wanted the minimum number of subbands (one) and a single frequency channel I could use:
+
+```
+psrdb toa download J1652-4838 --nchan 1 --minimum_nsubs
+```
+
+Which will download a `toa_J1652-4838_minimum_nsubs_nchan1.tim` file that is ready to be used by pulsar tools such as `tempo2`.

docs/index.rst

@@ -7,6 +7,7 @@ Welcome to psrdb's documentation!
 
    installation
    how_to_use
+   cli
    database_structure
    observation_ingest
    psrdb_modules

docs/installation.md

@@ -10,14 +10,20 @@ or if you don't use poetry, you can instead run
 pip install .
 ```
 
-To use the database you need a download token to read it or an ingest token to read and write/upload to it.
+To interact with the database you need to get an account on the [pulsars.org.au](https://pulsars.org.au) website.
+Once you have an account you then need to generate a token on this [page](https://pulsars.org.au/token_generate/).
 
-To get a read token run
+Set this token using the following command (you can put this in your `~/.bashrc`):
 ```
-get_token.sh
+export PSRDB_TOKEN=tokenhere
 ```
 
-The output token should be set as an environment variable using
+## Optional
+
+If you want to change the default url of https://pulsars.org.au/api/, you can change it by setting:
+
 ```
-export PSRDB_TOKEN=tokenhere
-export PSRDB_URL=https://pulsars.org.au/api/graphql/
+export PSRDB_URL=https://customurl.com/api/
+```
+
+It is likely that only developers will need to set this option.