diff --git a/.buildinfo b/.buildinfo index 61dfd87..f889448 100644 --- a/.buildinfo +++ b/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 878c152044dfb5049531fd6e897ecbff +config: 1b1501f97a74dce1b0ae5f6f0c31f428 tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/.doctrees/data_analysis/HPC-module-SLEAP.doctree b/.doctrees/data_analysis/HPC-module-SLEAP.doctree deleted file mode 100644 index 28a16e5..0000000 Binary files a/.doctrees/data_analysis/HPC-module-SLEAP.doctree and /dev/null differ diff --git a/.doctrees/data_analysis/index.doctree b/.doctrees/data_analysis/index.doctree index 5fe5fb6..9a6380c 100644 Binary files a/.doctrees/data_analysis/index.doctree and b/.doctrees/data_analysis/index.doctree differ diff --git a/.doctrees/environment.pickle b/.doctrees/environment.pickle index ca548e0..e466c8f 100644 Binary files a/.doctrees/environment.pickle and b/.doctrees/environment.pickle differ diff --git a/.doctrees/programming/Mount-ceph-ubuntu-temp.doctree b/.doctrees/programming/Mount-ceph-ubuntu-temp.doctree new file mode 100644 index 0000000..d2c8249 Binary files /dev/null and b/.doctrees/programming/Mount-ceph-ubuntu-temp.doctree differ diff --git a/.doctrees/programming/Mount-ceph-ubuntu.doctree b/.doctrees/programming/Mount-ceph-ubuntu.doctree index dd556f5..c6bf7a6 100644 Binary files a/.doctrees/programming/Mount-ceph-ubuntu.doctree and b/.doctrees/programming/Mount-ceph-ubuntu.doctree differ diff --git a/.doctrees/programming/SLURM-arguments.doctree b/.doctrees/programming/SLURM-arguments.doctree deleted file mode 100644 index 6ff0a3b..0000000 Binary files a/.doctrees/programming/SLURM-arguments.doctree and /dev/null differ diff --git a/.doctrees/programming/SSH-SWC-cluster.doctree b/.doctrees/programming/SSH-SWC-cluster.doctree index 39831b4..1aec041 100644 Binary files a/.doctrees/programming/SSH-SWC-cluster.doctree and b/.doctrees/programming/SSH-SWC-cluster.doctree differ diff --git a/.doctrees/programming/index.doctree b/.doctrees/programming/index.doctree index d318292..724141d 100644 Binary files a/.doctrees/programming/index.doctree and b/.doctrees/programming/index.doctree differ diff --git a/_sources/data_analysis/HPC-module-SLEAP.md.txt b/_sources/data_analysis/HPC-module-SLEAP.md.txt deleted file mode 100644 index 94378a8..0000000 --- a/_sources/data_analysis/HPC-module-SLEAP.md.txt +++ /dev/null @@ -1,598 +0,0 @@ -# Use the SLEAP module on the HPC cluster - -```{include} ../_static/swc-wiki-warning.md -``` - -```{include} ../_static/code-blocks-note.md -``` - -## Abbreviations -| Acronym | Meaning | -| ------- | -------------------------------------------- | -| SLEAP | Social LEAP Estimates Animal Poses | -| SWC | Sainsbury Wellcome Centre | -| HPC | High Performance Computing | -| SLURM | Simple Linux Utility for Resource Management | -| GUI | Graphical User Interface | - -## Prerequisites - -### Access to the HPC cluster -Verify that you can access HPC gateway node (typing your `` both times when prompted): -```{code-block} console -$ ssh @ssh.swc.ucl.ac.uk -$ ssh hpc-gw1 -``` -To learn more about accessing the HPC via SSH, see the [relevant how-to guide](ssh-cluster-target). - -### Access to the SLEAP module -Once you are on the HPC gateway node, SLEAP should be listed among the available modules when you run `module avail`: - -```{code-block} console -$ module avail -... -SLEAP/2023-03-13 -SLEAP/2023-08-01 -... -``` -- `SLEAP/2023-03-13` corresponds to `sleap v.1.2.9` -- `SLEAP/2023-08-01` corresponds to `sleap v.1.3.1` - -We recommend always using the latest version, which is the one loaded by default -when you run `module load SLEAP`. If you want to load a specific version, -you can do so by typing the full module name, -including the date e.g. `module load SLEAP/2023-03-13`. - -If a module has been successfully loaded, it will be listed when you run `module list`, -along with other modules it may depend on: - -```{code-block} console -$ module list -Currently Loaded Modulefiles: - 1) cuda/11.8 2) SLEAP/2023-08-01 -``` - -If you have troubles with loading the SLEAP module, -see this guide's [Troubleshooting section](#problems-with-the-sleap-module). - - -### Install SLEAP on your local PC/laptop -While you can delegate the GPU-intensive work to the HPC cluster, -you will need to use the SLEAP GUI for some steps, such as labelling frames. -Thus, you also need to install SLEAP on your local PC/laptop. - -We recommend following the official [SLEAP installation guide](https://sleap.ai/installation.html). If you already have `conda` installed, you may skip the `mamba` installation steps and opt for installing the `libmamba-solver` for `conda`: - -```{code-block} console -$ conda install -n base conda-libmamba-solver -$ conda config --set solver libmamba -``` -This will get you the much faster dependency resolution that `mamba` provides, without having to install `mamba` itself. -From `conda` version 23.10 onwards (released in November 2023), `libmamba-solver` [is anyway the default](https://conda.org/blog/2023-11-06-conda-23-10-0-release/). - -After that, you can follow the [rest of the SLEAP installation guide](https://sleap.ai/installation.html#conda-package), substituting `conda` for `mamba` in the relevant commands. - -::::{tab-set} - -:::{tab-item} Windows and Linux -```{code-block} console -$ conda create -y -n sleap -c conda-forge -c nvidia -c sleap -c anaconda sleap=1.3.1 -``` -::: - -:::{tab-item} MacOS X and Apple Silicon -```{code-block} console -$ conda create -y -n sleap -c conda-forge -c anaconda -c sleap sleap=1.3.1 -``` -::: - -:::: - -You may exchange `sleap=1.3.1` for other versions. To be on the safe side, ensure that your local installation version matches (or is at least close to) the one installed in the cluster module. - -### Mount the SWC filesystem on your local PC/laptop -The rest of this guide assumes that you have mounted the SWC filesystem on your local PC/laptop. -If you have not done so, please follow the relevant instructions on the -[SWC internal wiki](https://wiki.ucl.ac.uk/display/SSC/SWC+Storage+Platform+Overview). - -We will also assume that the data you are working with are stored in a `ceph` or `winstor` -directory to which you have access to. In the rest of this guide, we will use the path -`/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data` which contains a SLEAP project -for test purposes. You should replace this with the path to your own data. - -:::{dropdown} Data storage location matters -:color: warning -:icon: alert-fill - -The cluster has fast access to data stored on the `ceph` and `winstor` filesystems. -If your data is stored elsewhere, make sure to transfer it to `ceph` or `winstor` -before running the job. You can use tools such as [`rsync`](https://linux.die.net/man/1/rsync) -to copy data from your local machine to `ceph` via an ssh connection. For example: - -```{code-block} console -$ rsync -avz @ssh.swc.ucl.ac.uk:/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data -``` -::: - -## Model training -This will consist of two parts - [preparing a training job](#prepare-the-training-job) -(on your local SLEAP installation) and [running a training job](#run-the-training-job) -(on the HPC cluster's SLEAP module). Some evaluation metrics for the trained models -can be [viewed via the SLEAP GUI](#evaluate-the-trained-models) on your local SLEAP installation. - -### Prepare the training job -Follow the SLEAP instructions for [Creating a Project](https://sleap.ai/tutorials/new-project.html) -and [Initial Labelling](https://sleap.ai/tutorials/initial-labeling.html). -Ensure that the project file (e.g. `labels.v001.slp`) is saved in the mounted SWC filesystem -(as opposed to your local filesystem). - -Next, follow the instructions in [Remote Training](https://sleap.ai/guides/remote.html#remote-training), -i.e. *Predict* -> *Run Training…* -> *Export Training Job Package…*. -- For selecting the right configuration parameters, see [Configuring Models](https://sleap.ai/guides/choosing-models.html#) and [Troubleshooting Workflows](https://sleap.ai/guides/troubleshooting-workflows.html) -- Set the *Predict On* parameter to *nothing*. Remote training and inference (prediction) are easiest to run separately on the HPC Cluster. Also unselect *Visualize Predictions During Training* in training settings, if it's enabled by default. -- If you are working with camera view from above or below (as opposed to a side view), set the *Rotation Min Angle* and *Rotation Max Angle* to -180 and 180 respectively in the *Augmentation* section. -- Make sure to save the exported training job package (e.g. `labels.v001.slp.training_job.zip`) in the mounted SWC filesystem, for example, in the same directory as the project file. -- Unzip the training job package. This will create a folder with the same name (minus the `.zip` extension). This folder contains everything needed to run the training job on the HPC cluster. - -### Run the training job -Login to the HPC cluster as described above. -```{code-block} console -$ ssh @ssh.swc.ucl.ac.uk -$ ssh hpc-gw1 -``` -Navigate to the training job folder (replace with your own path) and list its contents: -```{code-block} console -:emphasize-lines: 12 -$ cd /ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data -$ cd labels.v001.slp.training_job -$ ls -1 -centered_instance.json -centroid.json -inference-script.sh -jobs.yaml -labels.v001.pkg.slp -labels.v001.slp.predictions.slp -train_slurm.sh -swc-hpc-pose-estimation -train-script.sh -``` -There should be a `train-script.sh` file created by SLEAP, which already contains the -commands to run the training. You can see the contents of the file by running `cat train-script.sh`: -```{code-block} bash -:caption: labels.v001.slp.training_job/train-script.sh -:name: train-script-sh -:linenos: -#!/bin/bash -sleap-train centroid.json labels.v001.pkg.slp -sleap-train centered_instance.json labels.v001.pkg.slp -``` -The precise commands will depend on the model configuration you chose in SLEAP. -Here we see two separate training calls, one for the 'centroid' and another for -the 'centered_instance' model. That's because in this example we have chosen -the ['Top-Down'](https://sleap.ai/tutorials/initial-training.html#training-options) -configuration, which consists of two neural networks - the first for isolating -the animal instances (by finding their centroids) and the second for predicting -all the body parts per instance. - -![Top-Down model configuration](https://sleap.ai/_images/topdown_approach.jpg) - -:::{dropdown} More on 'Top-Down' vs 'Bottom-Up' models -:color: info -:icon: info - -Although the 'Top-Down' configuration was designed with multiple animals in mind, -it can also be used for single-animal videos. It makes sense to use it for videos -where the animal occupies a relatively small portion of the frame - see -[Troubleshooting Workflows](https://sleap.ai/guides/troubleshooting-workflows.html) for more info. -::: - -Next you need to create a SLURM batch script, which will schedule the training job -on the HPC cluster. Create a new file called `train_slurm.sh` -(you can do this in the terminal with `nano`/`vim` or in a text editor of -your choice on your local PC/laptop). Here we create the script in the same folder -as the training job, but you can save it anywhere you want, or even keep track of it with `git`. - -```{code-block} console -$ nano train_slurm.sh -``` - -An example is provided below, followed by explanations. -```{code-block} bash -:caption: train_slurm.sh -:name: train-slurm-sh -:linenos: -#!/bin/bash - -#SBATCH -J slp_train # job name -#SBATCH -p gpu # partition (queue) -#SBATCH -N 1 # number of nodes -#SBATCH --mem 16G # memory pool for all cores -#SBATCH -n 4 # number of cores -#SBATCH -t 0-06:00 # time (D-HH:MM) -#SBATCH --gres gpu:1 # request 1 GPU (of any kind) -#SBATCH -o slurm.%x.%N.%j.out # STDOUT -#SBATCH -e slurm.%x.%N.%j.err # STDERR -#SBATCH --mail-type=ALL -#SBATCH --mail-user=user@domain.com - -# Load the SLEAP module -module load SLEAP - -# Define directories for SLEAP project and exported training job -SLP_DIR=/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data -SLP_JOB_NAME=labels.v001.slp.training_job -SLP_JOB_DIR=$SLP_DIR/$SLP_JOB_NAME - -# Go to the job directory -cd $SLP_JOB_DIR - -# Run the training script generated by SLEAP -./train-script.sh -``` - -In `nano`, you can save the file by pressing `Ctrl+O` and exit by pressing `Ctrl+X`. - -:::{dropdown} Explanation of the batch script -:color: info -:icon: info -- The `#SBATCH` lines are SLURM directives. They specify the resources needed -for the job, such as the number of nodes, CPUs, memory, etc. -A primer on the most useful SLURM arguments is provided in this [how-to guide](slurm-arguments-target). -For more information see the [SLURM documentation](https://slurm.schedmd.com/sbatch.html). - -- The `#` lines are comments. They are not executed by SLURM, but they are useful -for explaining the script to your future self and others. - -- The `module load SLEAP` line loads the latest SLEAP module and any other modules -it may depend on. - -- The `cd` line changes the working directory to the training job folder. -This is necessary because the `train-script.sh` file contains relative paths -to the model configuration and the project file. - -- The `./train-script.sh` line runs the training job (executes the contained commands). -::: - -:::{warning} -Before submitting the job, ensure that you have permissions to execute -both the batch script and the training script generated by SLEAP. -You can make these files executable by running in the terminal: - -```{code-block} console -$ chmod +x train-script.sh -$ chmod +x train_slurm.sh -``` - -If the scripts are not in your working directory, you will need to specify their full paths: - -```{code-block} console -$ chmod +x /path/to/train-script.sh -$ chmod +x /path/to/train_slurm.sh -``` -::: - -Now you can submit the batch script via running the following command -(in the same directory as the script): -```{code-block} console -$ sbatch train_slurm.sh -Submitted batch job 3445652 -``` - -You may monitor the progress of the job in various ways: - -::::{tab-set} - -:::{tab-item} squeue - -View the status of the queued/running jobs with [`squeue`](https://slurm.schedmd.com/squeue.html): - -```{code-block} console -$ squeue --me -JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) -3445652 gpu slp_train sirmpila R 23:11 1 gpu-sr670-20 -``` -::: - -:::{tab-item} sacct - -View status of running/completed jobs with [`sacct`](https://slurm.schedmd.com/sacct.html): - -```{code-block} console -$ sacct -JobID JobName Partition Account AllocCPUS State ExitCode ------------- ---------- ---------- ---------- ---------- ---------- -------- -3445652 slp_train gpu swc-ac 2 COMPLETED 0:0 -3445652.bat+ batch swc-ac 2 COMPLETED 0:0 -``` -Run `sacct` with some more helpful arguments. -For example, you can view jobs from the last 24 hours, displaying the time -elapsed and the peak memory usage in KB (MaxRSS): - -```{code-block} console -$ sacct \ - --starttime $(date -d '24 hours ago' +%Y-%m-%dT%H:%M:%S) \ - --endtime $(date +%Y-%m-%dT%H:%M:%S) \ - --format=JobID,JobName,Partition,State,Start,Elapsed,MaxRSS - -JobID JobName Partition State Start Elapsed MaxRSS ------------- ---------- ---------- ---------- ------------------- ---------- ---------- -4043595 slp_infer gpu FAILED 2023-10-10T18:14:31 00:00:35 -4043595.bat+ batch FAILED 2023-10-10T18:14:31 00:00:35 271104K -4043603 slp_infer gpu FAILED 2023-10-10T18:27:32 00:01:37 -4043603.bat+ batch FAILED 2023-10-10T18:27:32 00:01:37 423476K -4043611 slp_infer gpu PENDING Unknown 00:00:00 -``` -::: - -:::{tab-item} view the logs - -View the contents of standard output and error -(the node name and job ID will differ in each case): -```{code-block} console -$ cat slurm.gpu-sr670-20.3445652.out -$ cat slurm.gpu-sr670-20.3445652.err -``` -::: - -:::: - -### Evaluate the trained models -Upon successful completion of the training job, a `models` folder will have -been created in the training job directory. It contains one subfolder per -training run (by default prefixed with the date and time of the run). - -```{code-block} console -$ cd /ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data -$ cd labels.v001.slp.training_job -$ cd models -$ ls -1 -230509_141357.centered_instance -230509_141357.centroid -``` - -Each subfolder holds the trained model files (e.g. `best_model.h5`), -their configurations (`training_config.json`) and some evaluation metrics. - -```{code-block} console -$ cd 230509_141357.centered_instance -$ ls -1 -best_model.h5 -initial_config.json -labels_gt.train.slp -labels_gt.val.slp -labels_pr.train.slp -labels_pr.val.slp -metrics.train.npz -metrics.val.npz -training_config.json -training_log.csv -``` -The SLEAP GUI on your local machine can be used to quickly evaluate the trained models. - -- Select *Predict* -> *Evaluation Metrics for Trained Models...* -- Click on *Add Trained Models(s)* and select the folder containing the model(s) you want to evaluate. -- You can view the basic metrics on the shown table or you can also view a more detailed report (including plots) by clicking *View Metrics*. - -## Model inference -By inference, we mean using a trained model to predict the labels on new frames/videos. -SLEAP provides the `sleap-track` command line utility for running inference -on a single video or a folder of videos. - -Below is an example SLURM batch script that contains a `sleap-track` call. -```{code-block} bash -:caption: infer_slurm.sh -:name: infer-slurm-sh -:linenos: -#!/bin/bash - -#SBATCH -J slp_infer # job name -#SBATCH -p gpu # partition -#SBATCH -N 1 # number of nodes -#SBATCH --mem 16G # memory pool for all cores -#SBATCH -n 4 # number of cores -#SBATCH -t 0-02:00 # time (D-HH:MM) -#SBATCH --gres gpu:1 # request 1 GPU (of any kind) -#SBATCH -o slurm.%x.%N.%j.out # write STDOUT -#SBATCH -e slurm.%x.%N.%j.err # write STDERR -#SBATCH --mail-type=ALL -#SBATCH --mail-user=user@domain.com - -# Load the SLEAP module -module load SLEAP - -# Define directories for SLEAP project and exported training job -SLP_DIR=/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data -VIDEO_DIR=$SLP_DIR/videos -SLP_JOB_NAME=labels.v001.slp.training_job -SLP_JOB_DIR=$SLP_DIR/$SLP_JOB_NAME - -# Go to the job directory -cd $SLP_JOB_DIR -# Make a directory to store the predictions -mkdir -p predictions - -# Run the inference command -sleap-track $VIDEO_DIR/M708149_EPM_20200317_165049331-converted.mp4 \ - -m $SLP_JOB_DIR/models/231010_164307.centroid/training_config.json \ - -m $SLP_JOB_DIR/models/231010_164307.centered_instance/training_config.json \ - --gpu auto \ - --tracking.tracker simple \ - --tracking.similarity centroid \ - --tracking.post_connect_single_breaks 1 \ - -o predictions/labels.v001.slp.predictions.slp \ - --verbosity json \ - --no-empty-frames -``` -The script is very similar to the training script, with the following differences: -- The time limit `-t` is set lower, since inference is normally faster than training. This will however depend on the size of the video and the number of models used. -- The `./train-script.sh` line is replaced by the `sleap-track` command. -- The `\` character is used to split the long `sleap-track` command into multiple lines for readability. It is not necessary if the command is written on a single line. - -::: {dropdown} Explanation of the sleap-track arguments -:color: info -:icon: info - - Some important command line arguments are explained below. - You can view a full list of the available arguments by running `sleap-track --help`. -- The first argument is the path to the video file to be processed. -- The `-m` option is used to specify the path to the model configuration file(s) to be used for inference. In this example we use the two models that were trained above. -- The `--gpu` option is used to specify the GPU to be used for inference. The `auto` value will automatically select the GPU with the highest percentage of available memory (of the GPUs that are available on the machine/node) -- The options starting with `--tracking` specify parameters used for tracking the detected instances (animals) across frames. See SLEAP's guide on [tracking methods](https://sleap.ai/guides/proofreading.html#tracking-method-details) for more info. -- The `-o` option is used to specify the path to the output file containing the predictions. -- The above script will predict all the frames in the video. You may select specific frames via the `--frames` option. For example: `--frames 1-50` or `--frames 1,3,5,7,9`. -::: - -You can submit and monitor the inference job in the same way as the training job. -```{code-block} console -$ sbatch infer_slurm.sh -$ squeue --me -``` -Upon completion, a `labels.v001.slp.predictions.slp` file will have been created in the job directory. - -You can use the SLEAP GUI on your local machine to load and view the predictions: -*File* -> *Open Project...* -> select the `labels.v001.slp.predictions.slp` file. - -## The training-inference cycle -Now that you have some predictions, you can keep improving your models by repeating -the training-inference cycle. The basic steps are: -- Manually correct some of the predictions: see [Prediction-assisted labeling](https://sleap.ai/tutorials/assisted-labeling.html) -- Merge corrected labels into the initial training set: see [Merging guide](https://sleap.ai/guides/merging.html) -- Save the merged training set as `labels.v002.slp` -- Export a new training job `labels.v002.slp.training_job` (you may reuse the training configurations from `v001`) -- Repeat the training-inference cycle until satisfied - -## Troubleshooting - -### Problems with the SLEAP module - -In this section, we will describe how to test that the SLEAP module is loaded -correctly for you and that it can use the available GPUs. - -Login to the HPC cluster as described [above](#access-to-the-hpc-cluster). - -Start an interactive job on a GPU node. This step is necessary, because we need -to test the module's access to the GPU. -```{code-block} console -$ srun -p fast --gres=gpu:1 --pty bash -i -``` -:::{dropdown} Explain the above command -:color: info -:icon: info - -* `-p fast` requests a node from the 'fast' partition. This refers to the queue of nodes with a 3-hour time limit. They are meant for short jobs, such as testing. -* `--gres=gpu:1` requests 1 GPU of any kind -* `--pty` is short for 'pseudo-terminal'. -* The `-i` stands for 'interactive' - -Taken together, the above command will start an interactive bash terminal session -on a node of the 'fast' partition, equipped with 1 GPU. -::: - -First, let's verify that you are indeed on a node equipped with a functional -GPU, by typing `nvidia-smi`: -```{code-block} console -$ nvidia-smi -Wed Sep 27 10:34:35 2023 -+-----------------------------------------------------------------------------+ -| NVIDIA-SMI 525.125.06 Driver Version: 525.125.06 CUDA Version: 12.0 | -|-------------------------------+----------------------+----------------------+ -| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | -| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | -| | | MIG M. | -|===============================+======================+======================| -| 0 NVIDIA GeForce ... Off | 00000000:41:00.0 Off | N/A | -| 0% 42C P8 22W / 240W | 1MiB / 8192MiB | 0% Default | -| | | N/A | -+-------------------------------+----------------------+----------------------+ - -+-----------------------------------------------------------------------------+ -| Processes: | -| GPU GI CI PID Type Process name GPU Memory | -| ID ID Usage | -|=============================================================================| -| No running processes found | -+-----------------------------------------------------------------------------+ -``` -Your output should look similar to the above. You will be able to see the GPU -name, temperature, memory usage, etc. If you see an error message instead, -(even though you are on a GPU node) please contact the SWC Scientific Computing team. - -Next, load the SLEAP module. -```{code-block} console -$ module load SLEAP -Loading SLEAP/2023-08-01 - Loading requirement: cuda/11.8 -``` - -To verify that the module was loaded successfully: -```{code-block} console -$ module list -Currently Loaded Modulefiles: - 1) SLEAP/2023-08-01 -``` -You can essentially think of the module as a centrally installed conda environment. -When it is loaded, you should be using a particular Python executable. -You can verify this by running: - -```{code-block} console -$ which python -/ceph/apps/ubuntu-20/packages/SLEAP/2023-08-01/bin/python -``` - -Finally we will verify that the `sleap` python package can be imported and can -'see' the GPU. We will mostly just follow the -[relevant SLEAP instructions](https://sleap.ai/installation.html#testing-that-things-are-working). -First, start a Python interpreter: -```{code-block} console -$ python -``` -Next, run the following Python commands: - -::: {warning} -The `import sleap` command may take some time to run (more than a minute). -This is normal. Subsequent imports should be faster. -::: - -```{code-block} pycon ->>> import sleap - ->>> sleap.versions() -SLEAP: 1.3.1 -TensorFlow: 2.8.4 -Numpy: 1.21.6 -Python: 3.7.12 -OS: Linux-5.4.0-109-generic-x86_64-with-debian-bullseye-sid - ->>> sleap.system_summary() -GPUs: 1/1 available - Device: /physical_device:GPU:0 - Available: True - Initialized: False - Memory growth: None - ->>> import tensorflow as tf - ->>> print(tf.config.list_physical_devices('GPU')) -[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')] - ->>> tf.constant("Hello world!") - -``` - -If all is as expected, you can exit the Python interpreter, and then exit the GPU node -```{code-block} pycon ->>> exit() -``` -```{code-block} console -$ exit() -``` -If you encounter troubles with using the SLEAP module, contact the -Niko Sirmpilatze of the SWC [Neuroinformatics Unit](https://neuroinformatics.dev/). - -To completely exit the HPC cluster, you will need to logout of the SSH session twice: -```bash -$ logout -$ logout -``` -See [Set up SSH for the SWC HPC cluster](../programming/SSH-SWC-cluster.md) -for more information. diff --git a/_sources/data_analysis/index.md.txt b/_sources/data_analysis/index.md.txt index ce094e5..0e020c6 100644 --- a/_sources/data_analysis/index.md.txt +++ b/_sources/data_analysis/index.md.txt @@ -5,5 +5,4 @@ Guides related to the analysis of neuroscientific data, spanning a wide range of ```{toctree} :maxdepth: 1 -HPC-module-SLEAP ``` diff --git a/_sources/programming/Mount-ceph-ubuntu-temp.md.txt b/_sources/programming/Mount-ceph-ubuntu-temp.md.txt new file mode 100644 index 0000000..b6f7fbf --- /dev/null +++ b/_sources/programming/Mount-ceph-ubuntu-temp.md.txt @@ -0,0 +1,54 @@ +(target-mount-ceph-ubuntu-temp)= +# Mount ceph storage on Ubuntu temporarily +In this example, we will **temporarily** mount a partition of `ceph` in an Ubuntu machine. The mounting is temporary because it will only remain until the next reboot. To mount a partition of `ceph` permanently, see [Mount ceph storage on Ubuntu permanently](target-mount-ceph-ubuntu-perm). + + +## Prerequisites +- Administrator rights (`sudo`) on the local Ubuntu machine +- `cifs-utils` installed via `sudo apt-get install cifs-utils` + + +## Steps +### 1. Create a mount point +Create a directory to mount the storage to. Sensible places to do this on Ubuntu would be in `/mnt` or `/media`. In the example below, we will use `/mnt/ceph-neuroinformatics`. + +```bash +sudo mkdir /mnt/ceph-neuroinformatics +``` + +### 2. Mount the `ceph` partition +To mount the desired partition on the directory we just created, run the `mount` command with the appropriate options. In our example, this would be: +```bash +sudo mount -t cifs -o user=,domain=AD.SWC.UCL.AC.UK //ceph-gw02.hpc.swc.ucl.ac.uk/neuroinformatics /mnt/ceph-neuroinformatics +``` +:::{note} +You can check the full set of options for the `mount` command by running `mount --help` +::: + +Make sure to replace the following for your particular case: +- `//ceph-gw02.hpc.swc.ucl.ac.uk/neuroinformatics` with the path to your desired partition, e.g. `//ceph-gw02.hpc.swc.ucl.ac.uk/` +- `/media/ceph-neuroinformatics` with the path to the local mount point you created in the previous step. +- `` with your SWC username + +If the command is executed without errors you will be prompted for your SWC password. + +### 3. Check the partition is mounted correctly +It should show up in the list of mounting points when running `df -h` + +:::{note} +The command `df` prints out information about the file system +::: + +### 4. To unmount the storage +Run the following command +```bash +sudo umount /mnt/ceph-neuroinformatics +``` +Remember that because the mounting is temporary, the `ceph` partition will be unmounted upon rebooting our machine. + +You can check that the partition is correctly unmounted by running `df -h` and verifying it does not show in the output. + + +## References +- [Mount network drive under Linux temporarily/permanently](https://www.rz.uni-kiel.de/en/hints-howtos/connecting-a-network-share/Linux/through-temporary-permanent-mounting/mount-network-drive-under-linux-temporarily-permanently) +- [How to Mount a SMB Share in Ubuntu](https://support.zadarastorage.com/hc/en-us/articles/213024986-How-to-Mount-a-SMB-Share-in-Ubuntu) diff --git a/_sources/programming/Mount-ceph-ubuntu.md.txt b/_sources/programming/Mount-ceph-ubuntu.md.txt index 2a54eb2..a8cb02e 100644 --- a/_sources/programming/Mount-ceph-ubuntu.md.txt +++ b/_sources/programming/Mount-ceph-ubuntu.md.txt @@ -1,13 +1,18 @@ -# Mount ceph storage on Ubuntu +(target-mount-ceph-ubuntu-perm)= +# Mount ceph storage on Ubuntu permanently In this example, we will **permanently** mount the `neuroinformatics` partition of `ceph`. The same procedure can be used to mount other partitions, such as those belonging to particular labs or projects. +To mount a partition of `ceph` only temporarily, see [Mount ceph storage on Ubuntu temporarily](target-mount-ceph-ubuntu-temp). + + The following instructions have been tested on **Ubuntu 20.04 LTS**. ## Prerequisites - Administrator rights (`sudo`) on the local Ubuntu machine - `cifs-utils` installed via `sudo apt-get install cifs-utils` -## Store your SWC credentials +## Steps +### 1. Store your SWC credentials First, create a file to store your SWC login credentials and save it in your home directory. You can do that on the terminal via `nano ~/.smb_swc`. @@ -32,14 +37,14 @@ This will ensure the file is readable and writable only by the owner (you). However, if someone gets access to your machine with your user logged in, they will still be able to read the SWC password. ::: -## Create a mount point +### 2. Create a mount point Create a directory to mount the storage to. Sensible places to do this on Ubuntu would be in `/mnt` or `/media`. In the example below, we will use `/media/ceph-neuroinformatics`. ```bash sudo mkdir /media/ceph-neuroinformatics ``` -## Edit the fstab file +### 3. Edit the fstab file This will allow you to mount the storage automatically on boot. Before editing the file, make a backup of it, just in case. ```bash sudo cp /etc/fstab /etc/fstab.bak @@ -59,12 +64,12 @@ Make sure to replace the following: Save the file and exit. -## Mount the storage +### 4. Mount the storage You can now mount the storage by running `sudo mount -a`. If you get an error, check the `fstab` file for typos. If you get no error, you should be able to see the mounted storage by running `df -h`. The next time you reboot your machine, the storage should be mounted automatically. -## Unmount the storage +### 5. Unmount the storage To unmount the storage, run `sudo umount /media/ceph-neuroinformatics` (or whatever your mount point is). To permanently unmount the storage, remove the lines you added to the `fstab` file and run `sudo mount -a` again. diff --git a/_sources/programming/SLURM-arguments.md.txt b/_sources/programming/SLURM-arguments.md.txt deleted file mode 100644 index 6bfb81b..0000000 --- a/_sources/programming/SLURM-arguments.md.txt +++ /dev/null @@ -1,147 +0,0 @@ -(slurm-arguments-target)= -# SLURM arguments primer - -```{include} ../_static/swc-wiki-warning.md -``` - -## Abbreviations -| Acronym | Meaning | -| ------- | -------------------------------------------- | -| SWC | Sainsbury Wellcome Centre | -| HPC | High Performance Computing | -| SLURM | Simple Linux Utility for Resource Management | -| MPI | Message Passing Interface | - - -## Overview -SLURM is a job scheduler and resource manager used on the SWC HPC cluster. -It is responsible for allocating resources (e.g. CPU cores, GPUs, memory) to jobs submitted by users. -When submitting a job to SLURM, you can specify various arguments to request the resources you need. -These are called SLURM directives, and they are passed to SLURM via the `sbatch` or `srun` commands. - -These are often specified at the top of a SLURM job script, -e.g. the lines that start with `#SBATCH` in the following example: - -```{code-block} bash -#!/bin/bash - -#SBATCH -J my_job # job name -#SBATCH -p gpu # partition (queue) -#SBATCH -N 1 # number of nodes -#SBATCH --mem 16G # memory pool for all cores -#SBATCH -n 4 # number of cores -#SBATCH -t 0-06:00 # time (D-HH:MM) -#SBATCH --gres gpu:1 # request 1 GPU (of any kind) -#SBATCH -o slurm.%x.%N.%j.out # STDOUT -#SBATCH -e slurm.%x.%N.%j.err # STDERR -#SBATCH --mail-type=ALL -#SBATCH --mail-user=user@domain.com -#SBATCH --array=1-12%4 # job array index values - -# load modules -... - -# execute commands -... -``` -This guide provides only a brief overview of the most important SLURM arguments, -to demysify the above directives and help you get started with SLURM. -For a more detailed description see the [SLURM documentation](https://slurm.schedmd.com/sbatch.html). - -## Commonly used arguments - -### Partition (Queue) -- *Name:* `--partition` -- *Alias:* `-p` -- *Description:* Specifies the partition (or queue) to submit the job to. To see a list of all partitions/queues, the nodes they contain and their respective time limits, type `sinfo` when logged in to the HPC cluster. -- *Example values:* `gpu`, `cpu`, `fast`, `medium` - -### Job Name -- *Name:* `--job-name` -- *Alias:* `-J` -- *Description:* Specifies a name for the job, which will appear in various SLURM commands and logs, making it easier to identify the job (especially when you have multiple jobs queued up) -- *Example values:* `training_run_24` - -### Number of Nodes -- *Name:* `--nodes` -- *Alias:* `-N` -- *Description:* Defines the number of nodes required for the job. -- *Example values:* `1` - -:::{warning} -This should always be `1`, unless you really know what you're doing, -e.g. you are parallelising your code across multiple nodes with MPI. -::: - -### Number of Cores -- *Name:* `--ntasks` -- *Alias:* `-n` -- *Description:* Defines the number of cores (or tasks) required for the job. -- *Example values:* `4`, `8`, `16` - -### Memory Pool for All Cores -- *Name:* `--mem` -- *Description:* Specifies the total amount of memory (RAM) required for the job across all cores (per node) -- *Example values:* `8G`, `16G`, `32G` - -### Time Limit -- *Name:* `--time` -- *Alias:* `-t` -- *Description:* Sets the maximum time the job is allowed to run. The format is D-HH:MM, where D is days, HH is hours, and MM is minutes. -- *Example values:* `0-01:00` (1 hour), `0-04:00` (4 hours), `1-00:00` (1 day). - -:::{warning} -If the job exceeds the time limit, it will be terminated by SLURM. -On the other hand, avoid requesting way more time than what your job needs, -as this may delay its scheduling (depending on resource availability). -::: - -### Generic Resources (GPUs) -- *Name:* `--gres` -- *Description:* Requests generic resources, such as GPUs. -- *Example values:* `gpu:1`, `gpu:rtx2080:1`, `gpu:rtx5000:1`, `gpu:a100_2g.10gb:1` - -:::{warning} -No GPU will be allocated to you unless you specify it via the `--gres` argument (even if you are on the 'gpu' partition). -To request 1 GPU of any kind, use `--gres gpu:1`. To request a specific GPU type, you have to include its name, e.g. `--gres gpu:rtx2080:1`. -You can view the available GPU types on the [SWC internal wiki](https://wiki.ucl.ac.uk/display/SSC/CPU+and+GPU+Platform+architecture). -::: - -### Standard Output File -- *Name:* `--output` -- *Alias:* `-o` -- *Description:* Defines the file where the standard output (STDOUT) will be written. In the example script above, it's set to slurm.%x.%N.%j.out, where %x is the job name, %N is the node name and %j is the job ID. -- *Example values:* `slurm.%x.%N.%j.out`, `slurm.MyAwesomeJob.out` - -:::{note} -This file contains the output of the commands executed by the job (i.e. the messages that normally gets printed on the terminal). -::: - -### Standard Error File -- *Name:* `--error` -- *Alias:* `-e` -- *Description:* Specifies the file where the standard error (STDERR) will be written. In the example script above, it's set to slurm.%x.%N.%j.out, where %x is the job name, %N is the node name and %j is the job ID. -- *Example values:* `slurm.%N.%j.err`, `slurm.MyAwesomeJob.err` - -:::{note} -This file is very useful for debugging, as it contains all the error messages produced by the commands executed by the job. -::: - -### Email Notifications -- *Name:* `--mail-type` -- *Description:* Defines the conditions under which the user will be notified by email. -- *Example values:* `ALL`, `BEGIN`, `END`, `FAIL` - -### Email Address -- *Name:* `--mail-user` -- *Description:* Specifies the email address to which notifications will be sent. -- *Example values:* `user@domain.com` - -### Array jobs -- *Name:* `--array` -- *Description:* Job array index values (a list of integers in increasing order). The task index can be accessed via the `SLURM_ARRAY_TASK_ID` environment variable. -- *Example values:* `--array=1-10` (10 jobs), `--array=1-100%5` (100 jobs, but only 5 of them will be allowed to run in parallel at any given time). - -:::{warning} -If an array consists of many jobs, using the `%` syntax to limit the maximum number of parallel jobs is recommended to prevent overloading the cluster. -::: diff --git a/_sources/programming/SSH-SWC-cluster.md.txt b/_sources/programming/SSH-SWC-cluster.md.txt index 36e8241..549445c 100644 --- a/_sources/programming/SSH-SWC-cluster.md.txt +++ b/_sources/programming/SSH-SWC-cluster.md.txt @@ -1,4 +1,3 @@ -(ssh-cluster-target)= # Set up SSH for the SWC HPC cluster This guide explains how to connect to the SWC's HPC cluster via SSH. @@ -10,14 +9,14 @@ This guide explains how to connect to the SWC's HPC cluster via SSH. ``` ## Abbreviations -| Acronym | Meaning | -| ------- | -------------------------------------------- | -| SWC | Sainsbury Wellcome Centre | -| HPC | High Performance Computing | -| SLURM | Simple Linux Utility for Resource Management | -| SSH | Secure (Socket) Shell protocol | -| IDE | Integrated Development Environment | -| GUI | Graphical User Interface | +| Acronym | Meaning | +| --- | --- | +| SWC | Sainsbury Wellcome Centre | +| HPC | High Performance Computing | +| SLURM | Simple Linux Utility for Resource Management | +| SSH | Secure (Socket) Shell protocol | +| IDE | Integrated Development Environment | +| GUI | Graphical User Interface | ## Prerequisites - You have an SWC account and know your username and password. diff --git a/_sources/programming/index.md.txt b/_sources/programming/index.md.txt index 32fb11c..629588e 100644 --- a/_sources/programming/index.md.txt +++ b/_sources/programming/index.md.txt @@ -7,10 +7,10 @@ Small tips and tricks that do not warrant a long-form guide can be found in the ```{toctree} :maxdepth: 1 -SLURM-arguments SSH-SWC-cluster SSH-vscode Mount-ceph-ubuntu +Mount-ceph-ubuntu-temp Cookiecutter-cruft Troubleshooting ``` diff --git a/_static/check-solid.svg b/_static/check-solid.svg deleted file mode 100644 index 92fad4b..0000000 --- a/_static/check-solid.svg +++ /dev/null @@ -1,4 +0,0 @@ - - - - diff --git a/_static/clipboard.min.js b/_static/clipboard.min.js deleted file mode 100644 index 54b3c46..0000000 --- a/_static/clipboard.min.js +++ /dev/null @@ -1,7 +0,0 @@ -/*! - * clipboard.js v2.0.8 - * https://clipboardjs.com/ - * - * Licensed MIT © Zeno Rocha - */ -!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return o}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),c=n.n(e);function a(t){try{return document.execCommand(t)}catch(t){return}}var f=function(t){t=c()(t);return a("cut"),t};var l=function(t){var e,n,o,r=1 - - - - diff --git a/_static/copybutton.css b/_static/copybutton.css deleted file mode 100644 index f1916ec..0000000 --- a/_static/copybutton.css +++ /dev/null @@ -1,94 +0,0 @@ -/* Copy buttons */ -button.copybtn { - position: absolute; - display: flex; - top: .3em; - right: .3em; - width: 1.7em; - height: 1.7em; - opacity: 0; - transition: opacity 0.3s, border .3s, background-color .3s; - user-select: none; - padding: 0; - border: none; - outline: none; - border-radius: 0.4em; - /* The colors that GitHub uses */ - border: #1b1f2426 1px solid; - background-color: #f6f8fa; - color: #57606a; -} - -button.copybtn.success { - border-color: #22863a; - color: #22863a; -} - -button.copybtn svg { - stroke: currentColor; - width: 1.5em; - height: 1.5em; - padding: 0.1em; -} - -div.highlight { - position: relative; -} - -/* Show the copybutton */ -.highlight:hover button.copybtn, button.copybtn.success { - opacity: 1; -} - -.highlight button.copybtn:hover { - background-color: rgb(235, 235, 235); -} - -.highlight button.copybtn:active { - background-color: rgb(187, 187, 187); -} - -/** - * A minimal CSS-only tooltip copied from: - * https://codepen.io/mildrenben/pen/rVBrpK - * - * To use, write HTML like the following: - * - *

Short

- */ - .o-tooltip--left { - position: relative; - } - - .o-tooltip--left:after { - opacity: 0; - visibility: hidden; - position: absolute; - content: attr(data-tooltip); - padding: .2em; - font-size: .8em; - left: -.2em; - background: grey; - color: white; - white-space: nowrap; - z-index: 2; - border-radius: 2px; - transform: translateX(-102%) translateY(0); - transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); -} - -.o-tooltip--left:hover:after { - display: block; - opacity: 1; - visibility: visible; - transform: translateX(-100%) translateY(0); - transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); - transition-delay: .5s; -} - -/* By default the copy button shouldn't show up when printing a page */ -@media print { - button.copybtn { - display: none; - } -} diff --git a/_static/copybutton.js b/_static/copybutton.js deleted file mode 100644 index 39cbaa7..0000000 --- a/_static/copybutton.js +++ /dev/null @@ -1,248 +0,0 @@ -// Localization support -const messages = { - 'en': { - 'copy': 'Copy', - 'copy_to_clipboard': 'Copy to clipboard', - 'copy_success': 'Copied!', - 'copy_failure': 'Failed to copy', - }, - 'es' : { - 'copy': 'Copiar', - 'copy_to_clipboard': 'Copiar al portapapeles', - 'copy_success': '¡Copiado!', - 'copy_failure': 'Error al copiar', - }, - 'de' : { - 'copy': 'Kopieren', - 'copy_to_clipboard': 'In die Zwischenablage kopieren', - 'copy_success': 'Kopiert!', - 'copy_failure': 'Fehler beim Kopieren', - }, - 'fr' : { - 'copy': 'Copier', - 'copy_to_clipboard': 'Copier dans le presse-papier', - 'copy_success': 'Copié !', - 'copy_failure': 'Échec de la copie', - }, - 'ru': { - 'copy': 'Скопировать', - 'copy_to_clipboard': 'Скопировать в буфер', - 'copy_success': 'Скопировано!', - 'copy_failure': 'Не удалось скопировать', - }, - 'zh-CN': { - 'copy': '复制', - 'copy_to_clipboard': '复制到剪贴板', - 'copy_success': '复制成功!', - 'copy_failure': '复制失败', - }, - 'it' : { - 'copy': 'Copiare', - 'copy_to_clipboard': 'Copiato negli appunti', - 'copy_success': 'Copiato!', - 'copy_failure': 'Errore durante la copia', - } -} - -let locale = 'en' -if( document.documentElement.lang !== undefined - && messages[document.documentElement.lang] !== undefined ) { - locale = document.documentElement.lang -} - -let doc_url_root = DOCUMENTATION_OPTIONS.URL_ROOT; -if (doc_url_root == '#') { - doc_url_root = ''; -} - -/** - * SVG files for our copy buttons - */ -let iconCheck = ` - ${messages[locale]['copy_success']} - - -` - -// If the user specified their own SVG use that, otherwise use the default -let iconCopy = ``; -if (!iconCopy) { - iconCopy = ` - ${messages[locale]['copy_to_clipboard']} - - - -` -} - -/** - * Set up copy/paste for code blocks - */ - -const runWhenDOMLoaded = cb => { - if (document.readyState != 'loading') { - cb() - } else if (document.addEventListener) { - document.addEventListener('DOMContentLoaded', cb) - } else { - document.attachEvent('onreadystatechange', function() { - if (document.readyState == 'complete') cb() - }) - } -} - -const codeCellId = index => `codecell${index}` - -// Clears selected text since ClipboardJS will select the text when copying -const clearSelection = () => { - if (window.getSelection) { - window.getSelection().removeAllRanges() - } else if (document.selection) { - document.selection.empty() - } -} - -// Changes tooltip text for a moment, then changes it back -// We want the timeout of our `success` class to be a bit shorter than the -// tooltip and icon change, so that we can hide the icon before changing back. -var timeoutIcon = 2000; -var timeoutSuccessClass = 1500; - -const temporarilyChangeTooltip = (el, oldText, newText) => { - el.setAttribute('data-tooltip', newText) - el.classList.add('success') - // Remove success a little bit sooner than we change the tooltip - // So that we can use CSS to hide the copybutton first - setTimeout(() => el.classList.remove('success'), timeoutSuccessClass) - setTimeout(() => el.setAttribute('data-tooltip', oldText), timeoutIcon) -} - -// Changes the copy button icon for two seconds, then changes it back -const temporarilyChangeIcon = (el) => { - el.innerHTML = iconCheck; - setTimeout(() => {el.innerHTML = iconCopy}, timeoutIcon) -} - -const addCopyButtonToCodeCells = () => { - // If ClipboardJS hasn't loaded, wait a bit and try again. This - // happens because we load ClipboardJS asynchronously. - if (window.ClipboardJS === undefined) { - setTimeout(addCopyButtonToCodeCells, 250) - return - } - - // Add copybuttons to all of our code cells - const COPYBUTTON_SELECTOR = 'div.highlight pre'; - const codeCells = document.querySelectorAll(COPYBUTTON_SELECTOR) - codeCells.forEach((codeCell, index) => { - const id = codeCellId(index) - codeCell.setAttribute('id', id) - - const clipboardButton = id => - `` - codeCell.insertAdjacentHTML('afterend', clipboardButton(id)) - }) - -function escapeRegExp(string) { - return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string -} - -/** - * Removes excluded text from a Node. - * - * @param {Node} target Node to filter. - * @param {string} exclude CSS selector of nodes to exclude. - * @returns {DOMString} Text from `target` with text removed. - */ -function filterText(target, exclude) { - const clone = target.cloneNode(true); // clone as to not modify the live DOM - if (exclude) { - // remove excluded nodes - clone.querySelectorAll(exclude).forEach(node => node.remove()); - } - return clone.innerText; -} - -// Callback when a copy button is clicked. Will be passed the node that was clicked -// should then grab the text and replace pieces of text that shouldn't be used in output -function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { - var regexp; - var match; - - // Do we check for line continuation characters and "HERE-documents"? - var useLineCont = !!lineContinuationChar - var useHereDoc = !!hereDocDelim - - // create regexp to capture prompt and remaining line - if (isRegexp) { - regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') - } else { - regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') - } - - const outputLines = []; - var promptFound = false; - var gotLineCont = false; - var gotHereDoc = false; - const lineGotPrompt = []; - for (const line of textContent.split('\n')) { - match = line.match(regexp) - if (match || gotLineCont || gotHereDoc) { - promptFound = regexp.test(line) - lineGotPrompt.push(promptFound) - if (removePrompts && promptFound) { - outputLines.push(match[2]) - } else { - outputLines.push(line) - } - gotLineCont = line.endsWith(lineContinuationChar) & useLineCont - if (line.includes(hereDocDelim) & useHereDoc) - gotHereDoc = !gotHereDoc - } else if (!onlyCopyPromptLines) { - outputLines.push(line) - } else if (copyEmptyLines && line.trim() === '') { - outputLines.push(line) - } - } - - // If no lines with the prompt were found then just use original lines - if (lineGotPrompt.some(v => v === true)) { - textContent = outputLines.join('\n'); - } - - // Remove a trailing newline to avoid auto-running when pasting - if (textContent.endsWith("\n")) { - textContent = textContent.slice(0, -1) - } - return textContent -} - - -var copyTargetText = (trigger) => { - var target = document.querySelector(trigger.attributes['data-clipboard-target'].value); - - // get filtered text - let exclude = '.linenos, .gp, .go'; - - let text = filterText(target, exclude); - return formatCopyText(text, '', false, true, true, true, '', '') -} - - // Initialize with a callback so we can modify the text before copy - const clipboard = new ClipboardJS('.copybtn', {text: copyTargetText}) - - // Update UI with error/success messages - clipboard.on('success', event => { - clearSelection() - temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_success']) - temporarilyChangeIcon(event.trigger) - }) - - clipboard.on('error', event => { - temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_failure']) - }) -} - -runWhenDOMLoaded(addCopyButtonToCodeCells) \ No newline at end of file diff --git a/_static/copybutton_funcs.js b/_static/copybutton_funcs.js deleted file mode 100644 index dbe1aaa..0000000 --- a/_static/copybutton_funcs.js +++ /dev/null @@ -1,73 +0,0 @@ -function escapeRegExp(string) { - return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string -} - -/** - * Removes excluded text from a Node. - * - * @param {Node} target Node to filter. - * @param {string} exclude CSS selector of nodes to exclude. - * @returns {DOMString} Text from `target` with text removed. - */ -export function filterText(target, exclude) { - const clone = target.cloneNode(true); // clone as to not modify the live DOM - if (exclude) { - // remove excluded nodes - clone.querySelectorAll(exclude).forEach(node => node.remove()); - } - return clone.innerText; -} - -// Callback when a copy button is clicked. Will be passed the node that was clicked -// should then grab the text and replace pieces of text that shouldn't be used in output -export function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { - var regexp; - var match; - - // Do we check for line continuation characters and "HERE-documents"? - var useLineCont = !!lineContinuationChar - var useHereDoc = !!hereDocDelim - - // create regexp to capture prompt and remaining line - if (isRegexp) { - regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') - } else { - regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') - } - - const outputLines = []; - var promptFound = false; - var gotLineCont = false; - var gotHereDoc = false; - const lineGotPrompt = []; - for (const line of textContent.split('\n')) { - match = line.match(regexp) - if (match || gotLineCont || gotHereDoc) { - promptFound = regexp.test(line) - lineGotPrompt.push(promptFound) - if (removePrompts && promptFound) { - outputLines.push(match[2]) - } else { - outputLines.push(line) - } - gotLineCont = line.endsWith(lineContinuationChar) & useLineCont - if (line.includes(hereDocDelim) & useHereDoc) - gotHereDoc = !gotHereDoc - } else if (!onlyCopyPromptLines) { - outputLines.push(line) - } else if (copyEmptyLines && line.trim() === '') { - outputLines.push(line) - } - } - - // If no lines with the prompt were found then just use original lines - if (lineGotPrompt.some(v => v === true)) { - textContent = outputLines.join('\n'); - } - - // Remove a trailing newline to avoid auto-running when pasting - if (textContent.endsWith("\n")) { - textContent = textContent.slice(0, -1) - } - return textContent -} diff --git a/data_analysis/HPC-module-SLEAP.html b/data_analysis/HPC-module-SLEAP.html deleted file mode 100644 index c0b3585..0000000 --- a/data_analysis/HPC-module-SLEAP.html +++ /dev/null @@ -1,1149 +0,0 @@ - - - - - - - - - - - Use the SLEAP module on the HPC cluster — HowTo - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - -
-
-
-
- - - Ctrl+K -
-
- - - -
-
- -
- - - -
- - -
- -
- -
- - - -
- -
-
- -
- -
- -
- -
-
-
- - -
-
- - - - -
- -
- - -
-
- -
-
- -
- -
- - - - -
- -
- - -
-
- - - - - -
- -
-

Use the SLEAP module on the HPC cluster#

-
-

Warning

-

Some links within this document point to the -SWC internal wiki, -which is only accessible from within the SWC network.

-
-
- -Interpreting code blocks
-
-
-
-
-

Shell commands will be shown in code blocks like this -(with the $ sign indicating the shell prompt):

-
$ echo "Hello world!"
-
-
-

Similarly, Python code blocks will appear with the >>> sign indicating the -Python interpreter prompt:

-
>>> print("Hello world!")
-
-
-

The expected outputs of both Shell and Python commands will be shown without -any prompt:

-
Hello world!
-
-
-
-
-

Abbreviations#

- - - - - - - - - - - - - - - - - - - - - - - -

Acronym

Meaning

SLEAP

Social LEAP Estimates Animal Poses

SWC

Sainsbury Wellcome Centre

HPC

High Performance Computing

SLURM

Simple Linux Utility for Resource Management

GUI

Graphical User Interface

-
-
-

Prerequisites#

-
-

Access to the HPC cluster#

-

Verify that you can access HPC gateway node (typing your <SWC-PASSWORD> both times when prompted):

-
$ ssh <SWC-USERNAME>@ssh.swc.ucl.ac.uk
-$ ssh hpc-gw1
-
-
-

To learn more about accessing the HPC via SSH, see the relevant how-to guide.

-
-
-

Access to the SLEAP module#

-

Once you are on the HPC gateway node, SLEAP should be listed among the available modules when you run module avail:

-
$ module avail
-...
-SLEAP/2023-03-13
-SLEAP/2023-08-01
-...
-
-
-
    -

  • SLEAP/2023-03-13 corresponds to sleap v.1.2.9

    • -

  • SLEAP/2023-08-01 corresponds to sleap v.1.3.1

    • -
-

We recommend always using the latest version, which is the one loaded by default -when you run module load SLEAP. If you want to load a specific version, -you can do so by typing the full module name, -including the date e.g. module load SLEAP/2023-03-13.

-

If a module has been successfully loaded, it will be listed when you run module list, -along with other modules it may depend on:

-
$ module list
-Currently Loaded Modulefiles:
- 1) cuda/11.8   2) SLEAP/2023-08-01
-
-
-

If you have troubles with loading the SLEAP module, -see this guide’s Troubleshooting section.

-
-
-

Install SLEAP on your local PC/laptop#

-

While you can delegate the GPU-intensive work to the HPC cluster, -you will need to use the SLEAP GUI for some steps, such as labelling frames. -Thus, you also need to install SLEAP on your local PC/laptop.

-

We recommend following the official SLEAP installation guide. If you already have conda installed, you may skip the mamba installation steps and opt for installing the libmamba-solver for conda:

-
$ conda install -n base conda-libmamba-solver
-$ conda config --set solver libmamba
-
-
-

This will get you the much faster dependency resolution that mamba provides, without having to install mamba itself. -From conda version 23.10 onwards (released in November 2023), libmamba-solver is anyway the default.

-

After that, you can follow the rest of the SLEAP installation guide, substituting conda for mamba in the relevant commands.

-
- -
-
$ conda create -y -n sleap -c conda-forge -c nvidia -c sleap -c anaconda sleap=1.3.1
-
-
-
- -
-
$ conda create -y -n sleap -c conda-forge -c anaconda -c sleap sleap=1.3.1
-
-
-
-
-

You may exchange sleap=1.3.1 for other versions. To be on the safe side, ensure that your local installation version matches (or is at least close to) the one installed in the cluster module.

-
-
-

Mount the SWC filesystem on your local PC/laptop#

-

The rest of this guide assumes that you have mounted the SWC filesystem on your local PC/laptop. -If you have not done so, please follow the relevant instructions on the -SWC internal wiki.

-

We will also assume that the data you are working with are stored in a ceph or winstor -directory to which you have access to. In the rest of this guide, we will use the path -/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data which contains a SLEAP project -for test purposes. You should replace this with the path to your own data.

-
- -Data storage location matters
-
-
-
-
-

The cluster has fast access to data stored on the ceph and winstor filesystems. -If your data is stored elsewhere, make sure to transfer it to ceph or winstor -before running the job. You can use tools such as rsync -to copy data from your local machine to ceph via an ssh connection. For example:

-
$ rsync -avz <LOCAL-DIR> <SWC-USERNAME>@ssh.swc.ucl.ac.uk:/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data
-
-
-
-
-
-
-

Model training#

-

This will consist of two parts - preparing a training job -(on your local SLEAP installation) and running a training job -(on the HPC cluster’s SLEAP module). Some evaluation metrics for the trained models -can be viewed via the SLEAP GUI on your local SLEAP installation.

-
-

Prepare the training job#

-

Follow the SLEAP instructions for Creating a Project -and Initial Labelling. -Ensure that the project file (e.g. labels.v001.slp) is saved in the mounted SWC filesystem -(as opposed to your local filesystem).

-

Next, follow the instructions in Remote Training, -i.e. Predict -> Run Training… -> Export Training Job Package….

-
    -

  • For selecting the right configuration parameters, see Configuring Models and Troubleshooting Workflows

    • -

  • Set the Predict On parameter to nothing. Remote training and inference (prediction) are easiest to run separately on the HPC Cluster. Also unselect Visualize Predictions During Training in training settings, if it’s enabled by default.

    • -

  • If you are working with camera view from above or below (as opposed to a side view), set the Rotation Min Angle and Rotation Max Angle to -180 and 180 respectively in the Augmentation section.

    • -

  • Make sure to save the exported training job package (e.g. labels.v001.slp.training_job.zip) in the mounted SWC filesystem, for example, in the same directory as the project file.

    • -

  • Unzip the training job package. This will create a folder with the same name (minus the .zip extension). This folder contains everything needed to run the training job on the HPC cluster.

    • -
-
-
-

Run the training job#

-

Login to the HPC cluster as described above.

-
$ ssh <SWC-USERNAME>@ssh.swc.ucl.ac.uk
-$ ssh hpc-gw1
-
-
-

Navigate to the training job folder (replace with your own path) and list its contents:

-
$ cd /ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data
-$ cd labels.v001.slp.training_job
-$ ls -1
-centered_instance.json
-centroid.json
-inference-script.sh
-jobs.yaml
-labels.v001.pkg.slp
-labels.v001.slp.predictions.slp
-train_slurm.sh
-swc-hpc-pose-estimation
-train-script.sh
-
-
-

There should be a train-script.sh file created by SLEAP, which already contains the -commands to run the training. You can see the contents of the file by running cat train-script.sh:

-
-
labels.v001.slp.training_job/train-script.sh#
-
1#!/bin/bash
-2sleap-train centroid.json labels.v001.pkg.slp
-3sleap-train centered_instance.json labels.v001.pkg.slp
-
-
-
-

The precise commands will depend on the model configuration you chose in SLEAP. -Here we see two separate training calls, one for the ‘centroid’ and another for -the ‘centered_instance’ model. That’s because in this example we have chosen -the ‘Top-Down’ -configuration, which consists of two neural networks - the first for isolating -the animal instances (by finding their centroids) and the second for predicting -all the body parts per instance.

-

Top-Down model configuration

-
- -More on ‘Top-Down’ vs ‘Bottom-Up’ models
-
-
-
-
-

Although the ‘Top-Down’ configuration was designed with multiple animals in mind, -it can also be used for single-animal videos. It makes sense to use it for videos -where the animal occupies a relatively small portion of the frame - see -Troubleshooting Workflows for more info.

-
-

Next you need to create a SLURM batch script, which will schedule the training job -on the HPC cluster. Create a new file called train_slurm.sh -(you can do this in the terminal with nano/vim or in a text editor of -your choice on your local PC/laptop). Here we create the script in the same folder -as the training job, but you can save it anywhere you want, or even keep track of it with git.

-
$ nano train_slurm.sh
-
-
-

An example is provided below, followed by explanations.

-
-
train_slurm.sh#
-
 1#!/bin/bash
- 2
- 3#SBATCH -J slp_train # job name
- 4#SBATCH -p gpu # partition (queue)
- 5#SBATCH -N 1   # number of nodes
- 6#SBATCH --mem 16G # memory pool for all cores
- 7#SBATCH -n 4 # number of cores
- 8#SBATCH -t 0-06:00 # time (D-HH:MM)
- 9#SBATCH --gres gpu:1 # request 1 GPU (of any kind)
-10#SBATCH -o slurm.%x.%N.%j.out # STDOUT
-11#SBATCH -e slurm.%x.%N.%j.err # STDERR
-12#SBATCH --mail-type=ALL
-13#SBATCH --mail-user=user@domain.com
-14
-15# Load the SLEAP module
-16module load SLEAP
-17
-18# Define directories for SLEAP project and exported training job
-19SLP_DIR=/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data
-20SLP_JOB_NAME=labels.v001.slp.training_job
-21SLP_JOB_DIR=$SLP_DIR/$SLP_JOB_NAME
-22
-23# Go to the job directory
-24cd $SLP_JOB_DIR
-25
-26# Run the training script generated by SLEAP
-27./train-script.sh
-
-
-
-

In nano, you can save the file by pressing Ctrl+O and exit by pressing Ctrl+X.

-
- -Explanation of the batch script
-
-
-
-
-
    -

  • The #SBATCH lines are SLURM directives. They specify the resources needed -for the job, such as the number of nodes, CPUs, memory, etc. -A primer on the most useful SLURM arguments is provided in this how-to guide. -For more information see the SLURM documentation.

    • -

  • The # lines are comments. They are not executed by SLURM, but they are useful -for explaining the script to your future self and others.

    • -

  • The module load SLEAP line loads the latest SLEAP module and any other modules -it may depend on.

    • -

  • The cd line changes the working directory to the training job folder. -This is necessary because the train-script.sh file contains relative paths -to the model configuration and the project file.

    • -

  • The ./train-script.sh line runs the training job (executes the contained commands).

    • -
-
-
-

Warning

-

Before submitting the job, ensure that you have permissions to execute -both the batch script and the training script generated by SLEAP. -You can make these files executable by running in the terminal:

-
$ chmod +x train-script.sh
-$ chmod +x train_slurm.sh
-
-
-

If the scripts are not in your working directory, you will need to specify their full paths:

-
$ chmod +x /path/to/train-script.sh
-$ chmod +x /path/to/train_slurm.sh
-
-
-
-

Now you can submit the batch script via running the following command -(in the same directory as the script):

-
$ sbatch train_slurm.sh
-Submitted batch job 3445652
-
-
-

You may monitor the progress of the job in various ways:

-
- -
-

View the status of the queued/running jobs with squeue:

-
$ squeue --me
-JOBID    PARTITION  NAME     USER      ST  TIME   NODES  NODELIST(REASON)
-3445652  gpu        slp_train sirmpila  R   23:11  1      gpu-sr670-20
-
-
-
- -
-

View status of running/completed jobs with sacct:

-
$ sacct
-JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------- ---------- ---------- ---------- ---------- ---------- --------
-3445652      slp_train        gpu     swc-ac          2  COMPLETED      0:0
-3445652.bat+      batch                swc-ac          2  COMPLETED      0:0
-
-
-

Run sacct with some more helpful arguments. -For example, you can view jobs from the last 24 hours, displaying the time -elapsed and the peak memory usage in KB (MaxRSS):

-
$ sacct \
-  --starttime $(date -d '24 hours ago' +%Y-%m-%dT%H:%M:%S) \
-  --endtime $(date +%Y-%m-%dT%H:%M:%S) \
-  --format=JobID,JobName,Partition,State,Start,Elapsed,MaxRSS
-
-JobID           JobName  Partition      State               Start    Elapsed     MaxRSS
------------- ---------- ---------- ---------- ------------------- ---------- ----------
-4043595       slp_infer        gpu     FAILED 2023-10-10T18:14:31   00:00:35
-4043595.bat+      batch                FAILED 2023-10-10T18:14:31   00:00:35    271104K
-4043603       slp_infer        gpu     FAILED 2023-10-10T18:27:32   00:01:37
-4043603.bat+      batch                FAILED 2023-10-10T18:27:32   00:01:37    423476K
-4043611       slp_infer        gpu    PENDING             Unknown   00:00:00
-
-
-
- -
-

View the contents of standard output and error -(the node name and job ID will differ in each case):

-
$ cat slurm.gpu-sr670-20.3445652.out
-$ cat slurm.gpu-sr670-20.3445652.err
-
-
-
-
-
-
-

Evaluate the trained models#

-

Upon successful completion of the training job, a models folder will have -been created in the training job directory. It contains one subfolder per -training run (by default prefixed with the date and time of the run).

-
$ cd /ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data
-$ cd labels.v001.slp.training_job
-$ cd models
-$ ls -1
-230509_141357.centered_instance
-230509_141357.centroid
-
-
-

Each subfolder holds the trained model files (e.g. best_model.h5), -their configurations (training_config.json) and some evaluation metrics.

-
$ cd 230509_141357.centered_instance
-$ ls -1
-best_model.h5
-initial_config.json
-labels_gt.train.slp
-labels_gt.val.slp
-labels_pr.train.slp
-labels_pr.val.slp
-metrics.train.npz
-metrics.val.npz
-training_config.json
-training_log.csv
-
-
-

The SLEAP GUI on your local machine can be used to quickly evaluate the trained models.

-
    -

  • Select Predict -> Evaluation Metrics for Trained Models…

    • -

  • Click on Add Trained Models(s) and select the folder containing the model(s) you want to evaluate.

    • -

  • You can view the basic metrics on the shown table or you can also view a more detailed report (including plots) by clicking View Metrics.

    • -
-
-
-
-

Model inference#

-

By inference, we mean using a trained model to predict the labels on new frames/videos. -SLEAP provides the sleap-track command line utility for running inference -on a single video or a folder of videos.

-

Below is an example SLURM batch script that contains a sleap-track call.

-
-
infer_slurm.sh#
-
 1#!/bin/bash
- 2
- 3#SBATCH -J slp_infer # job name
- 4#SBATCH -p gpu # partition
- 5#SBATCH -N 1   # number of nodes
- 6#SBATCH --mem 16G # memory pool for all cores
- 7#SBATCH -n 4 # number of cores
- 8#SBATCH -t 0-02:00 # time (D-HH:MM)
- 9#SBATCH --gres gpu:1 # request 1 GPU (of any kind)
-10#SBATCH -o slurm.%x.%N.%j.out # write STDOUT
-11#SBATCH -e slurm.%x.%N.%j.err # write STDERR
-12#SBATCH --mail-type=ALL
-13#SBATCH --mail-user=user@domain.com
-14
-15# Load the SLEAP module
-16module load SLEAP
-17
-18# Define directories for SLEAP project and exported training job
-19SLP_DIR=/ceph/scratch/neuroinformatics-dropoff/SLEAP_HPC_test_data
-20VIDEO_DIR=$SLP_DIR/videos
-21SLP_JOB_NAME=labels.v001.slp.training_job
-22SLP_JOB_DIR=$SLP_DIR/$SLP_JOB_NAME
-23
-24# Go to the job directory
-25cd $SLP_JOB_DIR
-26# Make a directory to store the predictions
-27mkdir -p predictions
-28
-29# Run the inference command
-30sleap-track $VIDEO_DIR/M708149_EPM_20200317_165049331-converted.mp4 \
-31    -m $SLP_JOB_DIR/models/231010_164307.centroid/training_config.json \
-32    -m $SLP_JOB_DIR/models/231010_164307.centered_instance/training_config.json \
-33    --gpu auto \
-34    --tracking.tracker simple \
-35    --tracking.similarity centroid \
-36    --tracking.post_connect_single_breaks 1 \
-37    -o predictions/labels.v001.slp.predictions.slp \
-38    --verbosity json \
-39    --no-empty-frames
-
-
-
-

The script is very similar to the training script, with the following differences:

-
    -

  • The time limit -t is set lower, since inference is normally faster than training. This will however depend on the size of the video and the number of models used.

    • -

  • The ./train-script.sh line is replaced by the sleap-track command.

    • -

  • The \ character is used to split the long sleap-track command into multiple lines for readability. It is not necessary if the command is written on a single line.

    • -
-
- -Explanation of the sleap-track arguments
-
-
-
-
-

Some important command line arguments are explained below. -You can view a full list of the available arguments by running sleap-track --help.

-
    -

  • The first argument is the path to the video file to be processed.

    • -

  • The -m option is used to specify the path to the model configuration file(s) to be used for inference. In this example we use the two models that were trained above.

    • -

  • The --gpu option is used to specify the GPU to be used for inference. The auto value will automatically select the GPU with the highest percentage of available memory (of the GPUs that are available on the machine/node)

    • -

  • The options starting with --tracking specify parameters used for tracking the detected instances (animals) across frames. See SLEAP’s guide on tracking methods for more info.

    • -

  • The -o option is used to specify the path to the output file containing the predictions.

    • -

  • The above script will predict all the frames in the video. You may select specific frames via the --frames option. For example: --frames 1-50 or --frames 1,3,5,7,9.

    • -
-
-

You can submit and monitor the inference job in the same way as the training job.

-
$ sbatch infer_slurm.sh
-$ squeue --me
-
-
-

Upon completion, a labels.v001.slp.predictions.slp file will have been created in the job directory.

-

You can use the SLEAP GUI on your local machine to load and view the predictions: -File -> Open Project… -> select the labels.v001.slp.predictions.slp file.

-
-
-

The training-inference cycle#

-

Now that you have some predictions, you can keep improving your models by repeating -the training-inference cycle. The basic steps are:

-
    -

  • Manually correct some of the predictions: see Prediction-assisted labeling

    • -

  • Merge corrected labels into the initial training set: see Merging guide

    • -

  • Save the merged training set as labels.v002.slp

    • -

  • Export a new training job labels.v002.slp.training_job (you may reuse the training configurations from v001)

    • -

  • Repeat the training-inference cycle until satisfied

    • -
-
-
-

Troubleshooting#

-
-

Problems with the SLEAP module#

-

In this section, we will describe how to test that the SLEAP module is loaded -correctly for you and that it can use the available GPUs.

-

Login to the HPC cluster as described above.

-

Start an interactive job on a GPU node. This step is necessary, because we need -to test the module’s access to the GPU.

-
$ srun -p fast --gres=gpu:1 --pty bash -i
-
-
-
- -Explain the above command
-
-
-
-
-
    -

  • -p fast requests a node from the ‘fast’ partition. This refers to the queue of nodes with a 3-hour time limit. They are meant for short jobs, such as testing.

    • -

  • --gres=gpu:1 requests 1 GPU of any kind

    • -

  • --pty is short for ‘pseudo-terminal’.

    • -

  • The -i stands for ‘interactive’

    • -
-

Taken together, the above command will start an interactive bash terminal session -on a node of the ‘fast’ partition, equipped with 1 GPU.

-
-

First, let’s verify that you are indeed on a node equipped with a functional -GPU, by typing nvidia-smi:

-
$ nvidia-smi
-Wed Sep 27 10:34:35 2023
-+-----------------------------------------------------------------------------+
-| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
-|-------------------------------+----------------------+----------------------+
-| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
-| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
-|                               |                      |               MIG M. |
-|===============================+======================+======================|
-|   0  NVIDIA GeForce ...  Off  | 00000000:41:00.0 Off |                  N/A |
-|  0%   42C    P8    22W / 240W |      1MiB /  8192MiB |      0%      Default |
-|                               |                      |                  N/A |
-+-------------------------------+----------------------+----------------------+
-
-+-----------------------------------------------------------------------------+
-| Processes:                                                                  |
-|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
-|        ID   ID                                                   Usage      |
-|=============================================================================|
-|  No running processes found                                                 |
-+-----------------------------------------------------------------------------+
-
-
-

Your output should look similar to the above. You will be able to see the GPU -name, temperature, memory usage, etc. If you see an error message instead, -(even though you are on a GPU node) please contact the SWC Scientific Computing team.

-

Next, load the SLEAP module.

-
$ module load SLEAP
-Loading SLEAP/2023-08-01
-  Loading requirement: cuda/11.8
-
-
-

To verify that the module was loaded successfully:

-
$ module list
-Currently Loaded Modulefiles:
- 1) SLEAP/2023-08-01
-
-
-

You can essentially think of the module as a centrally installed conda environment. -When it is loaded, you should be using a particular Python executable. -You can verify this by running:

-
$ which python
-/ceph/apps/ubuntu-20/packages/SLEAP/2023-08-01/bin/python
-
-
-

Finally we will verify that the sleap python package can be imported and can -‘see’ the GPU. We will mostly just follow the -relevant SLEAP instructions. -First, start a Python interpreter:

-
$ python
-
-
-

Next, run the following Python commands:

-
-

Warning

-

The import sleap command may take some time to run (more than a minute). -This is normal. Subsequent imports should be faster.

-
-
>>> import sleap
-
->>> sleap.versions()
-SLEAP: 1.3.1
-TensorFlow: 2.8.4
-Numpy: 1.21.6
-Python: 3.7.12
-OS: Linux-5.4.0-109-generic-x86_64-with-debian-bullseye-sid
-
->>> sleap.system_summary()
-GPUs: 1/1 available
-  Device: /physical_device:GPU:0
-         Available: True
-        Initialized: False
-     Memory growth: None
-
->>> import tensorflow as tf
-
->>> print(tf.config.list_physical_devices('GPU'))
-[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
-
->>> tf.constant("Hello world!")
-<tf.Tensor: shape=(), dtype=string, numpy=b'Hello world!'>
-
-
-

If all is as expected, you can exit the Python interpreter, and then exit the GPU node

-
>>> exit()
-
-
-
$ exit()
-
-
-

If you encounter troubles with using the SLEAP module, contact the -Niko Sirmpilatze of the SWC Neuroinformatics Unit.

-

To completely exit the HPC cluster, you will need to logout of the SSH session twice:

-
$ logout
-$ logout
-
-
-

See Set up SSH for the SWC HPC cluster -for more information.

-
-
-
- - -
- - - - - - - -
- - - -
- -
-
- On this page -
-
- -
- - -
-
- -
- -
-
-
- - - - - - - - \ No newline at end of file diff --git a/data_analysis/index.html b/data_analysis/index.html index d0a0623..5231c93 100644 --- a/data_analysis/index.html +++ b/data_analysis/index.html @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - @@ -50,7 +47,7 @@ - + @@ -246,7 +243,7 @@
-
+
@@ -324,17 +321,6 @@
-
-
-
-
@@ -388,9 +374,6 @@

Data Analysis#

Guides related to the analysis of neuroscientific data, spanning a wide range of data types such as electrophysiology, behaviour, calcium imaging, histology, etc. The focus may be on the use of specific software tools, or on more general analysis tasks and concepts.

-
@@ -414,11 +397,11 @@

Data Analysis

next

-

Use the SLEAP module on the HPC cluster

+

Programming

 diff --git a/genindex.html b/genindex.html index f83ce6f..ca9991f 100644 --- a/genindex.html +++ b/genindex.html @@ -28,7 +28,6 @@ - @@ -40,8 +39,6 @@ - - diff --git a/index.html b/index.html index 2c873e7..c452a06 100644 --- a/index.html +++ b/index.html @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - diff --git a/objects.inv b/objects.inv index 8626f8e..ea0f3f0 100644 Binary files a/objects.inv and b/objects.inv differ diff --git a/open_science/Data-sharing.html b/open_science/Data-sharing.html index b3f579d..e154967 100644 --- a/open_science/Data-sharing.html +++ b/open_science/Data-sharing.html @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - diff --git a/open_science/Licensing.html b/open_science/Licensing.html index 6ef8524..751ee2d 100644 --- a/open_science/Licensing.html +++ b/open_science/Licensing.html @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - diff --git a/open_science/index.html b/open_science/index.html index 451962e..52ad4d8 100644 --- a/open_science/index.html +++ b/open_science/index.html @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - diff --git a/output.json b/output.json index 4f4abd7..6cd4790 100644 --- a/output.json +++ b/output.json @@ -1,70 +1,52 @@ {"filename": "_static/swc-wiki-warning.md", "lineno": 2, "status": "ignored", "code": 0, "uri": "https://wiki.ucl.ac.uk/display/SI/SWC+Intranet", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 94, "status": "ignored", "code": 0, "uri": "https://wiki.ucl.ac.uk/display/SSC/SWC+Storage+Platform+Overview", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 107, "status": "ignored", "code": 0, "uri": "https://linux.die.net/man/1/rsync", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 75, "status": "ignored", "code": 0, "uri": "https://neuromorpho.org/", "info": ""} {"filename": "open_science/Licensing.md", "lineno": 7, "status": "ignored", "code": 0, "uri": "https://www.uclb.com/", "info": ""} -{"filename": "programming/SLURM-arguments.md", "lineno": 105, "status": "ignored", "code": 0, "uri": "https://wiki.ucl.ac.uk/display/SSC/CPU+and+GPU+Platform+architecture", "info": ""} -{"filename": "programming/SSH-SWC-cluster.md", "lineno": 24, "status": "ignored", "code": 0, "uri": "https://wiki.ucl.ac.uk/display/SSC/High+Performance+Computing", "info": ""} -{"filename": "programming/SSH-SWC-cluster.md", "lineno": 24, "status": "ignored", "code": 0, "uri": "https://wiki.ucl.ac.uk/display/SSC/Logging+into+the+Cluster", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 26, "status": "unchecked", "code": 0, "uri": "#ssh-cluster-target", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 597, "status": "unchecked", "code": 0, "uri": "", "info": ""} +{"filename": "programming/Mount-ceph-ubuntu-temp.md", "lineno": 54, "status": "ignored", "code": 0, "uri": "https://support.zadarastorage.com/hc/en-us/articles/213024986-How-to-Mount-a-SMB-Share-in-Ubuntu", "info": ""} +{"filename": "programming/SSH-SWC-cluster.md", "lineno": 23, "status": "ignored", "code": 0, "uri": "https://wiki.ucl.ac.uk/display/SSC/High+Performance+Computing", "info": ""} +{"filename": "programming/SSH-SWC-cluster.md", "lineno": 23, "status": "ignored", "code": 0, "uri": "https://wiki.ucl.ac.uk/display/SSC/Logging+into+the+Cluster", "info": ""} +{"filename": "index.md", "lineno": 15, "status": "unchecked", "code": 0, "uri": "", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 26, "status": "unchecked", "code": 0, "uri": "#licensing-target", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 238, "status": "unchecked", "code": 0, "uri": "#slurm-arguments-target", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 64, "status": "working", "code": 0, "uri": "https://sleap.ai/installation.html", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 73, "status": "working", "code": 0, "uri": "https://sleap.ai/installation.html#conda-package", "info": ""} -{"filename": "open_science/Data-sharing.md", "lineno": 42, "status": "redirected", "code": 301, "uri": "http://www.brainimagelibrary.org/", "info": "https://www.brainimagelibrary.org/"} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 70, "status": "working", "code": 0, "uri": "https://conda.org/blog/2023-11-06-conda-23-10-0-release/", "info": ""} -{"filename": "open_science/Licensing.md", "lineno": 21, "status": "working", "code": 0, "uri": "https://choosealicense.com/", "info": ""} -{"filename": "programming/SSH-SWC-cluster.md", "lineno": 72, "status": "working", "code": 0, "uri": "https://code.visualstudio.com/", "info": ""} +{"filename": "programming/Mount-ceph-ubuntu-temp.md", "lineno": 3, "status": "unchecked", "code": 0, "uri": "#target-mount-ceph-ubuntu-perm", "info": ""} +{"filename": "programming/Mount-ceph-ubuntu.md", "lineno": 5, "status": "unchecked", "code": 0, "uri": "#target-mount-ceph-ubuntu-temp", "info": ""} {"filename": "programming/Cookiecutter-cruft.md", "lineno": 3, "status": "working", "code": 0, "uri": "https://cruft.github.io/cruft/", "info": ""} +{"filename": "open_science/Licensing.md", "lineno": 21, "status": "working", "code": 0, "uri": "https://choosealicense.com/", "info": ""} +{"filename": "open_science/Data-sharing.md", "lineno": 42, "status": "redirected", "code": 301, "uri": "http://www.brainimagelibrary.org/", "info": "https://www.brainimagelibrary.org/"} {"filename": "open_science/Data-sharing.md", "lineno": 43, "status": "working", "code": 0, "uri": "https://dandiarchive.org/", "info": ""} +{"filename": "programming/SSH-SWC-cluster.md", "lineno": 71, "status": "working", "code": 0, "uri": "https://code.visualstudio.com/", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 28, "status": "working", "code": 0, "uri": "https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 73, "status": "working", "code": 0, "uri": "https://crcns.org/", "info": ""} +{"filename": "open_science/Data-sharing.md", "lineno": 76, "status": "working", "code": 0, "uri": "http://ccdb.ucsd.edu/home", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 63, "status": "redirected", "code": 301, "uri": "https://datadryad.org", "info": "https://datadryad.org/stash"} {"filename": "open_science/Data-sharing.md", "lineno": 64, "status": "working", "code": 0, "uri": "https://gin.g-node.org/", "info": ""} -{"filename": "programming/SSH-SWC-cluster.md", "lineno": 34, "status": "working", "code": 0, "uri": "https://gitforwindows.org/", "info": ""} -{"filename": "open_science/Data-sharing.md", "lineno": 74, "status": "working", "code": 0, "uri": "https://gin.g-node.org/BrainGlobe/atlases", "info": ""} -{"filename": "open_science/Data-sharing.md", "lineno": 65, "status": "redirected", "code": 301, "uri": "https://ebrains.eu/service/share-data", "info": "https://www.ebrains.eu/data/share-data/"} {"filename": "open_science/Data-sharing.md", "lineno": 37, "status": "working", "code": 0, "uri": "https://gin.g-node.org/SainsburyWellcomeCentre", "info": ""} +{"filename": "programming/SSH-SWC-cluster.md", "lineno": 33, "status": "working", "code": 0, "uri": "https://gitforwindows.org/", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 26, "status": "working", "code": 0, "uri": "https://github.com/SainsburyWellcomeCentre", "info": ""} {"filename": "programming/Cookiecutter-cruft.md", "lineno": 3, "status": "working", "code": 0, "uri": "https://github.com/cruft/cruft", "info": ""} +{"filename": "open_science/Data-sharing.md", "lineno": 74, "status": "working", "code": 0, "uri": "https://gin.g-node.org/BrainGlobe/atlases", "info": ""} +{"filename": "programming/SSH-SWC-cluster.md", "lineno": 265, "status": "working", "code": 0, "uri": "https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 71, "status": "working", "code": 0, "uri": "https://nemoarchive.org/", "info": ""} -{"filename": "programming/SSH-SWC-cluster.md", "lineno": 266, "status": "working", "code": 0, "uri": "https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 49, "status": "working", "code": 0, "uri": "https://neuroblueprint.neuroinformatics.dev/", "info": ""} {"filename": "index.md", "lineno": 5, "status": "working", "code": 0, "uri": "https://neuroinformatics.dev", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 589, "status": "working", "code": 0, "uri": "https://neuroinformatics.dev/", "info": ""} -{"filename": "open_science/Data-sharing.md", "lineno": 72, "status": "working", "code": 0, "uri": "https://openneuro.org/", "info": ""} {"filename": "open_science/Licensing.md", "lineno": 51, "status": "working", "code": 0, "uri": "https://joinup.ec.europa.eu/collection/eupl/solution/joinup-licensing-assistant/jla-compatibility-checker", "info": ""} -{"filename": "open_science/Licensing.md", "lineno": 13, "status": "redirected", "code": 301, "uri": "https://opensource.org/licenses/BSD-3-Clause", "info": "https://opensource.org/license/bsd-3-clause/"} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 177, "status": "working", "code": 0, "uri": "https://sleap.ai/_images/topdown_approach.jpg", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 131, "status": "working", "code": 0, "uri": "https://sleap.ai/guides/choosing-models.html#", "info": ""} -{"filename": "open_science/Data-sharing.md", "lineno": 34, "status": "working", "code": 0, "uri": "https://rdr.ucl.ac.uk/The_Sainsbury_Wellcome_Centre", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 459, "status": "working", "code": 0, "uri": "https://sleap.ai/guides/merging.html", "info": ""} +{"filename": "open_science/Data-sharing.md", "lineno": 65, "status": "redirected", "code": 301, "uri": "https://ebrains.eu/service/share-data", "info": "https://www.ebrains.eu/data/share-data/"} +{"filename": "open_science/Licensing.md", "lineno": 7, "status": "working", "code": 0, "uri": "https://neuroinformatics.dev/", "info": ""} +{"filename": "open_science/Data-sharing.md", "lineno": 72, "status": "working", "code": 0, "uri": "https://openneuro.org/", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 77, "status": "redirected", "code": 301, "uri": "https://senselab.med.yale.edu/ModelDB/", "info": "https://modeldb.science/"} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 129, "status": "working", "code": 0, "uri": "https://sleap.ai/guides/remote.html#remote-training", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 542, "status": "working", "code": 0, "uri": "https://sleap.ai/installation.html#testing-that-things-are-working", "info": ""} -{"filename": "open_science/Data-sharing.md", "lineno": 76, "status": "working", "code": 0, "uri": "http://ccdb.ucsd.edu/home", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 131, "status": "working", "code": 0, "uri": "https://sleap.ai/guides/troubleshooting-workflows.html", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 458, "status": "working", "code": 0, "uri": "https://sleap.ai/tutorials/assisted-labeling.html", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 440, "status": "working", "code": 0, "uri": "https://sleap.ai/guides/proofreading.html#tracking-method-details", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 124, "status": "working", "code": 0, "uri": "https://sleap.ai/tutorials/new-project.html", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 169, "status": "working", "code": 0, "uri": "https://sleap.ai/tutorials/initial-training.html#training-options", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 124, "status": "working", "code": 0, "uri": "https://sleap.ai/tutorials/initial-labeling.html", "info": ""} +{"filename": "open_science/Data-sharing.md", "lineno": 34, "status": "working", "code": 0, "uri": "https://rdr.ucl.ac.uk/The_Sainsbury_Wellcome_Centre", "info": ""} {"filename": "programming/Troubleshooting.md", "lineno": 16, "status": "working", "code": 0, "uri": "https://support.github.com/contact?tags=rr-forks&subject=Detach%20Fork&flow=detach_fork", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 288, "status": "working", "code": 0, "uri": "https://slurm.schedmd.com/squeue.html", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 299, "status": "working", "code": 0, "uri": "https://slurm.schedmd.com/sacct.html", "info": ""} -{"filename": "data_analysis/HPC-module-SLEAP.md", "lineno": 238, "status": "working", "code": 0, "uri": "https://slurm.schedmd.com/sbatch.html", "info": ""} +{"filename": "open_science/Licensing.md", "lineno": 13, "status": "redirected", "code": 301, "uri": "https://opensource.org/licenses/BSD-3-Clause", "info": "https://opensource.org/license/bsd-3-clause/"} {"filename": "open_science/Licensing.md", "lineno": 21, "status": "redirected", "code": 301, "uri": "https://tldrlegal.com/", "info": "https://www.tldrlegal.com/"} +{"filename": "programming/SSH-SWC-cluster.md", "lineno": 272, "status": "working", "code": 0, "uri": "https://www.jetbrains.com/help/pycharm/remote-development-overview.html", "info": ""} +{"filename": "programming/SSH-SWC-cluster.md", "lineno": 272, "status": "working", "code": 0, "uri": "https://www.jetbrains.com/pycharm/", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 28, "status": "working", "code": 0, "uri": "https://www.doi.org/", "info": ""} -{"filename": "open_science/Data-sharing.md", "lineno": 68, "status": "working", "code": 0, "uri": "https://idr.openmicroscopy.org/", "info": ""} -{"filename": "programming/SSH-SWC-cluster.md", "lineno": 273, "status": "working", "code": 0, "uri": "https://www.jetbrains.com/help/pycharm/remote-development-overview.html", "info": ""} -{"filename": "programming/SSH-SWC-cluster.md", "lineno": 273, "status": "working", "code": 0, "uri": "https://www.jetbrains.com/pycharm/", "info": ""} -{"filename": "open_science/Licensing.md", "lineno": 54, "status": "working", "code": 0, "uri": "https://www.imperial.ac.uk/research-and-innovation/support-for-staff/scholarly-communication/research-data-management/sharing-data/research-software/", "info": ""} -{"filename": "open_science/Data-sharing.md", "lineno": 44, "status": "working", "code": 0, "uri": "https://www.ebi.ac.uk/bioimage-archive/", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 49, "status": "working", "code": 0, "uri": "https://www.nwb.org/", "info": ""} +{"filename": "open_science/Data-sharing.md", "lineno": 44, "status": "working", "code": 0, "uri": "https://www.ebi.ac.uk/bioimage-archive/", "info": ""} +{"filename": "open_science/Licensing.md", "lineno": 54, "status": "working", "code": 0, "uri": "https://www.imperial.ac.uk/research-and-innovation/support-for-staff/scholarly-communication/research-data-management/sharing-data/research-software/", "info": ""} +{"filename": "open_science/Licensing.md", "lineno": 7, "status": "working", "code": 0, "uri": "https://www.ucl.ac.uk/", "info": ""} +{"filename": "index.md", "lineno": 5, "status": "working", "code": 0, "uri": "https://www.sainsburywellcome.org/web/", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 66, "status": "working", "code": 0, "uri": "https://www.v2.opensourcebrain.org/", "info": ""} {"filename": "open_science/Data-sharing.md", "lineno": 54, "status": "unchecked", "code": 0, "uri": "mailto:adam.tyson@ucl.ac.uk", "info": ""} -{"filename": "index.md", "lineno": 5, "status": "working", "code": 0, "uri": "https://www.sainsburywellcome.org/web/", "info": ""} -{"filename": "open_science/Licensing.md", "lineno": 7, "status": "working", "code": 0, "uri": "https://www.ucl.ac.uk/", "info": ""} -{"filename": "open_science/Data-sharing.md", "lineno": 62, "status": "working", "code": 0, "uri": "https://zenodo.org/", "info": ""} +{"filename": "programming/Mount-ceph-ubuntu-temp.md", "lineno": 53, "status": "working", "code": 0, "uri": "https://www.rz.uni-kiel.de/en/hints-howtos/connecting-a-network-share/Linux/through-temporary-permanent-mounting/mount-network-drive-under-linux-temporarily-permanently", "info": ""} {"filename": "index.md", "lineno": 5, "status": "working", "code": 0, "uri": "https://www.ucl.ac.uk/gatsby/gatsby-computational-neuroscience-unit", "info": ""} +{"filename": "open_science/Data-sharing.md", "lineno": 62, "status": "working", "code": 0, "uri": "https://zenodo.org/", "info": ""} +{"filename": "open_science/Data-sharing.md", "lineno": 68, "status": "working", "code": 0, "uri": "https://idr.openmicroscopy.org/", "info": ""} diff --git a/output.txt b/output.txt index 43f0f40..62d367d 100644 --- a/output.txt +++ b/output.txt @@ -1,6 +1,6 @@ open_science/Data-sharing.md:42: [redirected permanently] http://www.brainimagelibrary.org/ to https://www.brainimagelibrary.org/ open_science/Data-sharing.md:63: [redirected permanently] https://datadryad.org to https://datadryad.org/stash open_science/Data-sharing.md:65: [redirected permanently] https://ebrains.eu/service/share-data to https://www.ebrains.eu/data/share-data/ -open_science/Licensing.md:13: [redirected permanently] https://opensource.org/licenses/BSD-3-Clause to https://opensource.org/license/bsd-3-clause/ open_science/Data-sharing.md:77: [redirected permanently] https://senselab.med.yale.edu/ModelDB/ to https://modeldb.science/ +open_science/Licensing.md:13: [redirected permanently] https://opensource.org/licenses/BSD-3-Clause to https://opensource.org/license/bsd-3-clause/ open_science/Licensing.md:21: [redirected permanently] https://tldrlegal.com/ to https://www.tldrlegal.com/ diff --git a/programming/Cookiecutter-cruft.html b/programming/Cookiecutter-cruft.html index ae3df7f..d5f081a 100644 --- a/programming/Cookiecutter-cruft.html +++ b/programming/Cookiecutter-cruft.html @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - @@ -51,7 +48,7 @@ - + @@ -333,10 +330,10 @@ aria-label="Section Navigation">

Section Navigation

@@ -450,12 +447,12 @@

Commit#

previous

-

Mount ceph storage on Ubuntu

+

Mount ceph storage on Ubuntu temporarily

 - SLURM arguments primer — HowTo + Mount ceph storage on Ubuntu temporarily — HowTo @@ -29,7 +29,6 @@ - @@ -41,17 +40,15 @@ - - - - + + - - + + @@ -333,10 +330,10 @@ aria-label="Section Navigation">

Section Navigation

@@ -379,7 +376,7 @@
  • Programming
    • -
  • SLURM...
    • +
  • Mount ceph...
    • @@ -396,206 +393,67 @@
    -
    -

    SLURM arguments primer#

    -
    -

    Warning

    -

    Some links within this document point to the -SWC internal wiki, -which is only accessible from within the SWC network.

    -
    -
    -

    Abbreviations#

    - - - - - - - - - - - - - - - - - - - - -

    Acronym

    Meaning

    SWC

    Sainsbury Wellcome Centre

    HPC

    High Performance Computing

    SLURM

    Simple Linux Utility for Resource Management

    MPI

    Message Passing Interface

    -
    -
    -

    Overview#

    -

    SLURM is a job scheduler and resource manager used on the SWC HPC cluster. -It is responsible for allocating resources (e.g. CPU cores, GPUs, memory) to jobs submitted by users. -When submitting a job to SLURM, you can specify various arguments to request the resources you need. -These are called SLURM directives, and they are passed to SLURM via the sbatch or srun commands.

    -

    These are often specified at the top of a SLURM job script, -e.g. the lines that start with #SBATCH in the following example:

    -
    #!/bin/bash
-
-#SBATCH -J my_job # job name
-#SBATCH -p gpu # partition (queue)
-#SBATCH -N 1   # number of nodes
-#SBATCH --mem 16G # memory pool for all cores
-#SBATCH -n 4 # number of cores
-#SBATCH -t 0-06:00 # time (D-HH:MM)
-#SBATCH --gres gpu:1 # request 1 GPU (of any kind)
-#SBATCH -o slurm.%x.%N.%j.out # STDOUT
-#SBATCH -e slurm.%x.%N.%j.err # STDERR
-#SBATCH --mail-type=ALL
-#SBATCH --mail-user=user@domain.com
-#SBATCH --array=1-12%4 # job array index values
-
-# load modules
-...
-
-# execute commands
-...
-
    -
    -

    This guide provides only a brief overview of the most important SLURM arguments, -to demysify the above directives and help you get started with SLURM. -For a more detailed description see the SLURM documentation.

    -
    -
    -

    Commonly used arguments#

    -
    -

    Partition (Queue)#

    -
      -

    • Name: --partition

      • -

    • Alias: -p

      • -

    • Description: Specifies the partition (or queue) to submit the job to. To see a list of all partitions/queues, the nodes they contain and their respective time limits, type sinfo when logged in to the HPC cluster.

      • -

    • Example values: gpu, cpu, fast, medium

      • -
    -
    -
    -

    Job Name#

    -
      -

    • Name: --job-name

      • -

    • Alias: -J

      • -

    • Description: Specifies a name for the job, which will appear in various SLURM commands and logs, making it easier to identify the job (especially when you have multiple jobs queued up)

      • -

    • Example values: training_run_24

      • -
    -
    -
    -

    Number of Nodes#

    -
      -

    • Name: --nodes

      • -

    • Alias: -N

      • -

    • Description: Defines the number of nodes required for the job.

      • -

    • Example values: 1

      • -
    -
    -

    Warning

    -

    This should always be 1, unless you really know what you’re doing, -e.g. you are parallelising your code across multiple nodes with MPI.

    -
    -
    -
    -

    Number of Cores#

    -
      -

    • Name: --ntasks

      • -

    • Alias: -n

      • -

    • Description: Defines the number of cores (or tasks) required for the job.

      • -

    • Example values: 4, 8, 16

      • -
    -
    -
    -

    Memory Pool for All Cores#

    +
    +

    Mount ceph storage on Ubuntu temporarily#

    +

    In this example, we will temporarily mount a partition of ceph in an Ubuntu machine. The mounting is temporary because it will only remain until the next reboot. To mount a partition of ceph permanently, see Mount ceph storage on Ubuntu permanently.

    +
    +

    Prerequisites#

      -

    • Name: --mem

      • -

    • Description: Specifies the total amount of memory (RAM) required for the job across all cores (per node)

      • -

    • Example values: 8G, 16G, 32G

      • +

    • Administrator rights (sudo) on the local Ubuntu machine

      • +

    • cifs-utils installed via sudo apt-get install cifs-utils

    -
    -

    Time Limit#

    -
      -

    • Name: --time

      • -

    • Alias: -t

      • -

    • Description: Sets the maximum time the job is allowed to run. The format is D-HH:MM, where D is days, HH is hours, and MM is minutes.

      • -

    • Example values: 0-01:00 (1 hour), 0-04:00 (4 hours), 1-00:00 (1 day).

      • -
    -
    -

    Warning

    -

    If the job exceeds the time limit, it will be terminated by SLURM. -On the other hand, avoid requesting way more time than what your job needs, -as this may delay its scheduling (depending on resource availability).

    +
    +

    Steps#

    +
    +

    1. Create a mount point#

    +

    Create a directory to mount the storage to. Sensible places to do this on Ubuntu would be in /mnt or /media. In the example below, we will use /mnt/ceph-neuroinformatics.

    +
    sudo mkdir /mnt/ceph-neuroinformatics
+
    -
    -

    Generic Resources (GPUs)#

    -
      -

    • Name: --gres

      • -

    • Description: Requests generic resources, such as GPUs.

      • -

    • Example values: gpu:1, gpu:rtx2080:1, gpu:rtx5000:1, gpu:a100_2g.10gb:1

      • -
    -
    -

    Warning

    -

    No GPU will be allocated to you unless you specify it via the --gres argument (even if you are on the ‘gpu’ partition). -To request 1 GPU of any kind, use --gres gpu:1. To request a specific GPU type, you have to include its name, e.g. --gres gpu:rtx2080:1. -You can view the available GPU types on the SWC internal wiki.

    +
    +

    2. Mount the ceph partition#

    +

    To mount the desired partition on the directory we just created, run the mount command with the appropriate options. In our example, this would be:

    +
    sudo mount -t cifs -o user=<SWC-USERNAME>,domain=AD.SWC.UCL.AC.UK //ceph-gw02.hpc.swc.ucl.ac.uk/neuroinformatics /mnt/ceph-neuroinformatics
+
    -
    -
    -

    Standard Output File#

    -
      -

    • Name: --output

      • -

    • Alias: -o

      • -

    • Description: Defines the file where the standard output (STDOUT) will be written. In the example script above, it’s set to slurm.%x.%N.%j.out, where %x is the job name, %N is the node name and %j is the job ID.

      • -

    • Example values: slurm.%x.%N.%j.out, slurm.MyAwesomeJob.out

      • -

    Note

    -

    This file contains the output of the commands executed by the job (i.e. the messages that normally gets printed on the terminal).

    +

    You can check the full set of options for the mount command by running mount --help

    -
    -
    -

    Standard Error File#

    +

    Make sure to replace the following for your particular case:

      -

    • Name: --error

      • -

    • Alias: -e

      • -

    • Description: Specifies the file where the standard error (STDERR) will be written. In the example script above, it’s set to slurm.%x.%N.%j.out, where %x is the job name, %N is the node name and %j is the job ID.

      • -

    • Example values: slurm.%N.%j.err, slurm.MyAwesomeJob.err

      • +

    • //ceph-gw02.hpc.swc.ucl.ac.uk/neuroinformatics with the path to your desired partition, e.g. //ceph-gw02.hpc.swc.ucl.ac.uk/<LAB-NAME>

      • +

    • /media/ceph-neuroinformatics with the path to the local mount point you created in the previous step.

      • +

    • <SWC-USERNAME> with your SWC username

    +

    If the command is executed without errors you will be prompted for your SWC password.

    +
    +
    +

    3. Check the partition is mounted correctly#

    +

    It should show up in the list of mounting points when running df -h

    Note

    -

    This file is very useful for debugging, as it contains all the error messages produced by the commands executed by the job.

    +

    The command df prints out information about the file system

    -
    -

    Email Notifications#

    -
      -

    • Name: --mail-type

      • -

    • Description: Defines the conditions under which the user will be notified by email.

      • -

    • Example values: ALL, BEGIN, END, FAIL

      • -
    +
    +

    4. To unmount the storage#

    +

    Run the following command

    +
    sudo umount /mnt/ceph-neuroinformatics
+
    +
    +

    Remember that because the mounting is temporary, the ceph partition will be unmounted upon rebooting our machine.

    +

    You can check that the partition is correctly unmounted by running df -h and verifying it does not show in the output.

    -
    -

    Email Address#

    -
      -

    • Name: --mail-user

      • -

    • Description: Specifies the email address to which notifications will be sent.

      • -

    • Example values: user@domain.com

      • -
    -
    -

    Array jobs#

    +
    +

    References#

    -
    -

    Warning

    -

    If an array consists of many jobs, using the % syntax to limit the maximum number of parallel jobs is recommended to prevent overloading the cluster.

    -
    -
    @@ -610,20 +468,20 @@

    Array jobs

    previous

    -

    Programming

    +

    Mount ceph storage on Ubuntu permanently

    next

    -

    Set up SSH for the SWC HPC cluster

    +

    Update cookiecutter-generated files with cruft

     @@ -642,23 +500,15 @@

    Array jobs

    diff --git a/programming/Mount-ceph-ubuntu.html b/programming/Mount-ceph-ubuntu.html index 65882aa..37a9145 100644 --- a/programming/Mount-ceph-ubuntu.html +++ b/programming/Mount-ceph-ubuntu.html @@ -8,7 +8,7 @@ - Mount ceph storage on Ubuntu — HowTo + Mount ceph storage on Ubuntu permanently — HowTo @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - @@ -50,7 +47,7 @@ - + @@ -333,10 +330,10 @@ aria-label="Section Navigation">

    Section Navigation

    @@ -396,9 +393,10 @@
    -
    -

    Mount ceph storage on Ubuntu#

    +
    +

    Mount ceph storage on Ubuntu permanently#

    In this example, we will permanently mount the neuroinformatics partition of ceph. The same procedure can be used to mount other partitions, such as those belonging to particular labs or projects.

    +

    To mount a partition of ceph only temporarily, see Mount ceph storage on Ubuntu temporarily.

    The following instructions have been tested on Ubuntu 20.04 LTS.

    Prerequisites#

    @@ -407,8 +405,10 @@

    Prerequisitescifs-utils installed via sudo apt-get install cifs-utils

    +
    +

    Steps#

    -

    Store your SWC credentials#

    +

    1. Store your SWC credentials#

    First, create a file to store your SWC login credentials and save it in your home directory. You can do that on the terminal via nano ~/.smb_swc.

    The .smb_swc file contents should look like this:

    @@ -431,14 +431,14 @@

    Store your SWC credentials -

    Create a mount point#

    +

    2. Create a mount point#

    Create a directory to mount the storage to. Sensible places to do this on Ubuntu would be in /mnt or /media. In the example below, we will use /media/ceph-neuroinformatics.

    sudo mkdir /media/ceph-neuroinformatics
    -

    Edit the fstab file#

    +

    3. Edit the fstab file#

    This will allow you to mount the storage automatically on boot. Before editing the file, make a backup of it, just in case.

    sudo cp /etc/fstab /etc/fstab.bak
    @@ -459,15 +459,16 @@

    Edit the fstab file -

    Mount the storage#

    +

    4. Mount the storage#

    You can now mount the storage by running sudo mount -a. If you get an error, check the fstab file for typos. If you get no error, you should be able to see the mounted storage by running df -h.

    The next time you reboot your machine, the storage should be mounted automatically.

    -

    Unmount the storage#

    +

    5. Unmount the storage#

    To unmount the storage, run sudo umount /media/ceph-neuroinformatics (or whatever your mount point is).

    To permanently unmount the storage, remove the lines you added to the fstab file and run sudo mount -a again.

    +
    @@ -490,11 +491,11 @@

    Unmount the storage

    next

    -

    Update cookiecutter-generated files with cruft

    +

    Mount ceph storage on Ubuntu temporarily

     @@ -514,11 +515,14 @@

    Unmount the storage

    diff --git a/programming/SSH-SWC-cluster.html b/programming/SSH-SWC-cluster.html index 2e2686b..23302f6 100644 --- a/programming/SSH-SWC-cluster.html +++ b/programming/SSH-SWC-cluster.html @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - @@ -51,7 +48,7 @@ - + @@ -333,10 +330,10 @@ aria-label="Section Navigation">

    Section Navigation

    @@ -397,7 +394,7 @@
    -

    Set up SSH for the SWC HPC cluster#

    +

    Set up SSH for the SWC HPC cluster#

    This guide explains how to connect to the SWC’s HPC cluster via SSH.

    Warning

    @@ -709,12 +706,12 @@

    Remote development

    previous

    -

    SLURM arguments primer

    +

    Programming

     - @@ -41,8 +40,6 @@ - - @@ -50,7 +47,7 @@ - + @@ -329,10 +326,10 @@ aria-label="Section Navigation">

    Section Navigation

    @@ -450,7 +447,7 @@

    SSH into an unmanaged machine using VSCode

    next

    -

    Mount ceph storage on Ubuntu

    +

    Mount ceph storage on Ubuntu permanently

     diff --git a/programming/Troubleshooting.html b/programming/Troubleshooting.html index 30a3247..cdf1ee9 100644 --- a/programming/Troubleshooting.html +++ b/programming/Troubleshooting.html @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - @@ -333,10 +330,10 @@ aria-label="Section Navigation">

    Section Navigation

    diff --git a/programming/index.html b/programming/index.html index bbff924..778c409 100644 --- a/programming/index.html +++ b/programming/index.html @@ -29,7 +29,6 @@ - @@ -41,8 +40,6 @@ - - @@ -50,8 +47,8 @@ - - + + @@ -329,10 +326,10 @@ aria-label="Section Navigation">

    Section Navigation

    @@ -395,10 +392,10 @@

    ProgrammingTroubleshooting section.

    @@ -416,20 +413,20 @@

    Programming

    previous

    -

    Use the SLEAP module on the HPC cluster

    +

    Data Analysis

    next

    -

    SLURM arguments primer

    +

    Set up SSH for the SWC HPC cluster

     diff --git a/search.html b/search.html index b3a4ecb..206e639 100644 --- a/search.html +++ b/search.html @@ -27,7 +27,6 @@ - @@ -39,8 +38,6 @@ - - diff --git a/searchindex.js b/searchindex.js index c7f8e92..2f3fe90 100644 --- a/searchindex.js +++ b/searchindex.js @@ -1 +1 @@ -Search.setIndex({"docnames": ["data_analysis/HPC-module-SLEAP", "data_analysis/index", "index", "open_science/Data-sharing", "open_science/Licensing", "open_science/index", "programming/Cookiecutter-cruft", "programming/Mount-ceph-ubuntu", "programming/SLURM-arguments", "programming/SSH-SWC-cluster", "programming/SSH-vscode", "programming/Troubleshooting", "programming/index"], "filenames": ["data_analysis/HPC-module-SLEAP.md", "data_analysis/index.md", "index.md", "open_science/Data-sharing.md", "open_science/Licensing.md", "open_science/index.md", "programming/Cookiecutter-cruft.md", "programming/Mount-ceph-ubuntu.md", "programming/SLURM-arguments.md", "programming/SSH-SWC-cluster.md", "programming/SSH-vscode.md", "programming/Troubleshooting.md", "programming/index.md"], "titles": ["Use the SLEAP module on the HPC cluster", "Data Analysis", "Neuroinformatics Unit HowTo", "Data Sharing", "Software licensing", "Open Science", "Update cookiecutter-generated files with cruft", "Mount ceph storage on Ubuntu", "SLURM arguments primer", "Set up SSH for the SWC HPC cluster", "SSH into an unmanaged machine using VSCode", "Troubleshooting", "Programming"], "terms": {"some": [0, 3, 8, 9], "link": [0, 4, 8, 9], "within": [0, 3, 8, 9], "thi": [0, 2, 3, 6, 7, 8, 9, 10, 11], "document": [0, 4, 8, 9], "point": [0, 6, 8, 9], "intern": [0, 8, 9], "wiki": [0, 8, 9], "which": [0, 4, 7, 8, 9], "i": [0, 2, 3, 6, 7, 8, 9, 10], "onli": [0, 4, 6, 7, 8, 9], "from": [0, 3, 6, 8, 9, 10, 11], "network": [0, 8, 9, 10], "interpret": [0, 9], "code": [0, 2, 8, 9, 12], "block": [0, 9], "shell": [0, 9], "command": [0, 6, 7, 8, 9, 12], "shown": [0, 9], "like": [0, 7, 9], "sign": [0, 9], "indic": [0, 9], "prompt": [0, 6, 9], "echo": [0, 9], "hello": [0, 9], "world": [0, 9], "similarli": [0, 9], "python": [0, 4, 6, 9], "appear": [0, 8, 9, 11], "print": [0, 8, 9], "expect": [0, 3, 9], "output": [0, 9], "both": [0, 9], "without": [0, 3, 4, 9, 11], "ani": [0, 3, 4, 8, 9], "acronym": [0, 8, 9], "mean": [0, 8, 9], "social": 0, "leap": 0, "estim": 0, "anim": 0, "pose": 0, "sainsburi": [0, 2, 4, 8, 9], "wellcom": [0, 2, 3, 4, 8, 9], "centr": [0, 2, 4, 8, 9], "high": [0, 8, 9], "perform": [0, 8, 9], "comput": [0, 2, 3, 4, 8, 9], "slurm": [0, 9, 12], "simpl": [0, 4, 8, 9], "linux": [0, 8, 9, 10, 12], "util": [0, 3, 7, 8, 9], "resourc": [0, 3, 4, 9], "manag": [0, 8, 9], "gui": [0, 9], "graphic": [0, 9], "user": [0, 3, 7, 8, 9, 10], "interfac": [0, 4, 8, 9], "verifi": [0, 9], "you": [0, 2, 3, 4, 6, 7, 8, 9, 10, 11], "can": [0, 2, 3, 6, 7, 8, 9, 10, 11, 12], "gatewai": [0, 9], "node": [0, 3, 9], "type": [0, 1, 3, 8, 9], "password": [0, 7, 9, 10], "time": [0, 3, 4, 6, 7, 9], "when": [0, 6, 8, 9, 10, 11], "ssh": [0, 12], "usernam": [0, 7, 9], "ucl": [0, 3, 4, 7, 9, 10], "ac": [0, 3, 7, 9, 10], "uk": [0, 3, 7, 9, 10], "gw1": [0, 9], "To": [0, 3, 7, 8, 9], "learn": [0, 9], "more": [0, 1, 6, 8, 9], "about": [0, 3, 4, 6, 9], "via": [0, 7, 8, 9, 11], "see": [0, 6, 7, 8, 9], "relev": [0, 3], "how": [0, 2, 9], "guid": [0, 1, 2, 4, 5, 8, 9, 12], "onc": [0, 6, 9], "ar": [0, 2, 3, 4, 6, 7, 8, 9, 10], "should": [0, 2, 3, 4, 6, 7, 8, 9], "list": [0, 4, 8, 9], "among": 0, "avail": [0, 3, 8, 9], "2023": 0, "03": 0, "13": 0, "08": 0, "01": [0, 8], "correspond": 0, "v": [0, 6, 9], "1": [0, 8, 9], "2": 0, "9": 0, "3": [0, 4, 11], "we": [0, 3, 4, 6, 7], "recommend": [0, 3, 4, 8, 9], "alwai": [0, 4, 8], "latest": 0, "version": [0, 3, 4, 6, 12], "one": [0, 4, 9, 10], "load": [0, 8], "default": [0, 9], "If": [0, 3, 6, 7, 8, 9, 10, 11], "want": [0, 4, 6, 9, 11], "specif": [0, 1, 3, 4, 8], "do": [0, 3, 6, 7, 8, 10, 12], "so": [0, 9], "full": 0, "name": [0, 4, 7, 9, 11], "includ": [0, 3, 8, 9], "date": [0, 6], "e": [0, 3, 6, 7, 8, 9], "g": [0, 3, 7, 8, 9], "ha": [0, 4, 6, 9], "been": [0, 6, 7], "successfulli": [0, 9], "along": [0, 9], "other": [0, 3, 7, 8, 9], "mai": [0, 1, 3, 4, 8, 9], "depend": [0, 8], "current": [0, 3, 11], "modulefil": 0, "cuda": 0, "11": 0, "8": [0, 8], "have": [0, 3, 6, 7, 8, 9, 10], "troubl": 0, "": [0, 4, 6, 8, 9, 11], "section": [0, 3, 9, 12], "while": 0, "deleg": 0, "gpu": [0, 9], "intens": [0, 9], "work": [0, 4, 9, 10], "need": [0, 6, 8, 9], "step": [0, 4, 7, 9], "label": 0, "frame": 0, "thu": 0, "also": [0, 4, 6, 9], "follow": [0, 4, 7, 8, 9, 10, 11], "offici": [0, 4], "alreadi": [0, 9], "conda": [0, 11], "skip": 0, "mamba": 0, "opt": 0, "libmamba": 0, "solver": 0, "n": [0, 8], "base": [0, 6], "config": [0, 10], "set": [0, 6, 8, 10, 12], "get": [0, 6, 7, 8, 9, 11], "much": 0, "faster": 0, "resolut": 0, "provid": [0, 3, 6, 8, 9], "itself": 0, "23": 0, "10": [0, 8, 9, 11], "onward": 0, "releas": 0, "novemb": 0, "anywai": 0, "after": [0, 6], "rest": 0, "substitut": 0, "window": [0, 9, 10], "creat": [0, 3, 6, 9, 10], "y": [0, 6], "c": [0, 9], "forg": 0, "nvidia": 0, "anaconda": 0, "maco": [0, 9], "x": [0, 8, 9], "appl": 0, "silicon": 0, "exchang": 0, "safe": 0, "side": 0, "ensur": [0, 3, 4, 7], "match": 0, "least": 0, "close": 0, "assum": [0, 9], "done": [0, 6, 7, 9], "pleas": [0, 3, 4, 6], "instruct": [0, 7, 11], "data": [0, 2, 5], "store": 0, "ceph": [0, 9, 12], "winstor": 0, "directori": [0, 7, 9], "In": [0, 3, 4, 6, 7, 8, 9, 10], "path": [0, 7, 9], "scratch": [0, 3], "neuroinformat": [0, 3, 4, 6, 7], "dropoff": 0, "sleap_hpc_test_data": 0, "contain": [0, 6, 8, 9], "project": [0, 3, 7], "test": [0, 7], "purpos": [0, 3], "replac": [0, 7, 10], "own": [0, 3, 4], "storag": [0, 12], "locat": [0, 9], "matter": 0, "fast": [0, 8], "elsewher": [0, 3], "make": [0, 3, 6, 7, 8, 9, 10], "sure": [0, 6, 7, 10], "transfer": [0, 4], "befor": [0, 3, 4, 6, 7, 9], "tool": [0, 1, 2, 4, 9, 10], "rsync": 0, "copi": [0, 4, 9], "machin": [0, 7, 9, 12], "an": [0, 3, 4, 7, 8, 9, 11, 12], "connect": [0, 9, 10], "For": [0, 3, 4, 6, 7, 8, 9], "exampl": [0, 3, 6, 7, 8, 9, 10], "avz": 0, "dir": 0, "consist": [0, 8], "two": [0, 9], "part": [0, 4], "metric": 0, "view": [0, 3, 8, 9], "initi": [0, 3], "file": [0, 3, 4, 10, 11, 12], "v001": 0, "slp": 0, "save": [0, 7, 9], "oppos": 0, "next": [0, 7, 9, 10], "remot": [0, 10], "predict": 0, "export": 0, "packag": [0, 4, 6], "select": [0, 4, 9], "right": [0, 7], "configur": [0, 9, 10], "paramet": 0, "workflow": 0, "On": [0, 2, 8, 9, 10], "noth": 0, "easiest": 0, "separ": [0, 4], "unselect": 0, "visual": [0, 9], "dure": 0, "enabl": [0, 3], "camera": 0, "abov": [0, 4, 7, 8, 9], "below": [0, 7, 9], "rotat": 0, "min": 0, "angl": 0, "max": 0, "180": 0, "respect": [0, 8], "augment": 0, "training_job": 0, "zip": 0, "same": [0, 6, 7, 9], "unzip": 0, "folder": [0, 3, 9, 10], "minu": 0, "extens": [0, 9, 10], "everyth": [0, 6], "login": [0, 7], "describ": [0, 9], "navig": [0, 9], "its": [0, 6, 8], "content": [0, 6, 7, 9], "cd": [0, 9, 10], "l": [0, 9], "centered_inst": 0, "json": [0, 6], "centroid": 0, "script": [0, 8, 9], "sh": 0, "yaml": 0, "pkg": 0, "train_slurm": 0, "There": [0, 3, 4, 9], "cat": 0, "bin": [0, 8, 11], "bash": [0, 8, 9], "precis": 0, "chose": 0, "here": [0, 9], "call": [0, 4, 8], "anoth": [0, 9], "That": [0, 9], "becaus": [0, 6, 9], "chosen": 0, "top": [0, 8], "down": 0, "neural": 0, "first": [0, 6, 7, 9], "isol": 0, "instanc": [0, 3], "find": [0, 2, 4, 7, 9, 10], "second": [0, 9], "all": [0, 3, 6, 9, 11], "bodi": 0, "per": [0, 8], "bottom": [0, 9], "up": [0, 4, 6, 8, 12], "although": 0, "wa": [0, 4], "design": 0, "multipl": [0, 8, 9], "mind": 0, "singl": [0, 3, 9], "video": 0, "It": [0, 4, 6, 8, 9], "sens": 0, "where": [0, 8, 9], "occupi": 0, "rel": 0, "small": [0, 2, 3, 11, 12], "portion": 0, "info": 0, "batch": [0, 9], "schedul": [0, 8], "new": [0, 4, 6, 7, 9], "termin": [0, 7, 8, 9], "nano": [0, 7, 9, 11], "vim": 0, "text": [0, 4, 7, 9], "editor": [0, 7, 9], "choic": 0, "anywher": 0, "even": [0, 8], "keep": 0, "track": [0, 6], "git": [0, 3, 6, 9], "explan": 0, "sbatch": [0, 8, 9], "j": [0, 8], "slp_train": 0, "p": [0, 8], "partit": [0, 7], "queue": 0, "number": 0, "mem": [0, 8], "16g": [0, 8], "memori": 0, "pool": 0, "core": 0, "4": [0, 8], "t": [0, 6, 8, 9], "0": [0, 7, 8], "06": [0, 8], "00": [0, 8], "d": [0, 8], "hh": [0, 8], "mm": [0, 8], "gre": [0, 8], "request": [0, 8], "kind": [0, 8], "o": [0, 8, 9], "out": [0, 7, 8, 9], "stdout": [0, 8], "err": [0, 8], "stderr": [0, 8], "mail": [0, 8], "domain": [0, 3, 7, 8], "com": [0, 4, 6, 8], "defin": [0, 8, 9], "slp_dir": 0, "slp_job_nam": 0, "slp_job_dir": 0, "go": 0, "gener": [0, 1, 2, 3, 9, 12], "press": [0, 3, 6, 9], "ctrl": [0, 9, 11], "exit": [0, 7, 9, 11], "line": [0, 7, 8, 9, 12], "direct": [0, 8, 9], "thei": [0, 3, 4, 6, 7, 8, 9], "specifi": [0, 6, 8, 9], "cpu": [0, 8], "etc": [0, 1, 3, 7, 9, 12], "A": [0, 5, 9], "primer": [0, 12], "most": [0, 2, 4, 8, 9], "argument": [0, 9, 12], "inform": [0, 2, 3, 4], "comment": 0, "execut": [0, 8], "explain": [0, 9], "futur": [0, 6], "self": 0, "chang": [0, 6, 7, 11], "necessari": 0, "submit": [0, 8, 9], "permiss": [0, 4, 7], "chmod": [0, 7], "now": [0, 6, 7, 9], "3445652": 0, "monitor": [0, 9], "progress": 0, "variou": [0, 8], "wai": [0, 3, 4, 8, 10], "squeue": 0, "statu": 0, "queu": [0, 8], "me": 0, "jobid": 0, "st": 0, "nodelist": 0, "reason": [0, 4], "sirmpila": 0, "r": 0, "sr670": [0, 9], "20": [0, 7], "sacct": 0, "complet": 0, "jobnam": 0, "account": [0, 3, 9], "alloccpu": 0, "state": 0, "exitcod": 0, "bat": 0, "help": [0, 3, 4, 8], "last": [0, 9], "24": [0, 10], "hour": [0, 8], "displai": [0, 9], "elaps": 0, "peak": 0, "usag": 0, "kb": 0, "maxrss": 0, "starttim": 0, "ago": 0, "m": 0, "dt": 0, "h": [0, 7], "endtim": 0, "format": [0, 3, 7, 8], "start": [0, 3, 8, 9], "4043595": 0, "slp_infer": 0, "fail": [0, 6, 8], "10t18": 0, "14": 0, "31": 0, "35": 0, "271104k": 0, "4043603": 0, "27": 0, "32": 0, "37": 0, "423476k": 0, "4043611": 0, "pend": 0, "unknown": 0, "log": [0, 7, 8], "standard": [0, 3], "error": [0, 6, 7], "id": [0, 7, 8, 9, 12], "differ": [0, 4, 9], "each": [0, 6], "case": [0, 3, 4, 6, 7, 9], "upon": [0, 4], "success": 0, "subfold": 0, "prefix": 0, "230509_141357": 0, "hold": 0, "best_model": 0, "h5": 0, "training_config": 0, "initial_config": 0, "labels_gt": 0, "val": 0, "labels_pr": 0, "npz": 0, "training_log": 0, "csv": 0, "quickli": 0, "click": [0, 9, 10], "add": [0, 7, 9], "basic": [0, 9], "tabl": 0, "detail": [0, 4, 8], "report": [0, 3], "plot": 0, "By": [0, 9], "infer_slurm": 0, "02": 0, "write": [0, 4], "video_dir": 0, "mkdir": [0, 7], "m708149_epm_20200317_165049331": 0, "convert": 0, "mp4": 0, "231010_164307": 0, "auto": 0, "tracker": 0, "similar": [0, 4, 9], "post_connect_single_break": 0, "verbos": 0, "empti": 0, "veri": [0, 8, 9], "limit": 0, "lower": 0, "sinc": [0, 6, 9], "normal": [0, 4, 8], "than": [0, 8], "howev": [0, 3, 4, 7, 9], "size": 0, "charact": 0, "split": 0, "long": [0, 2, 5, 9, 12], "readabl": [0, 7], "written": [0, 4, 8], "import": [0, 4, 8], "process": [0, 6, 9], "option": [0, 9], "were": [0, 6], "valu": [0, 8], "automat": [0, 7, 9], "highest": 0, "percentag": 0, "detect": 0, "across": [0, 8, 9], "method": 0, "50": 0, "5": [0, 8], "7": 0, "open": [0, 2, 3, 4, 6, 7, 9, 10], "improv": 0, "repeat": [0, 6], "manual": [0, 6], "correct": [0, 6], "assist": [0, 11], "merg": [0, 6], "v002": 0, "reus": [0, 3], "until": 0, "satisfi": [0, 6], "correctli": 0, "interact": [0, 9], "srun": [0, 8, 9], "pty": 0, "refer": 0, "meant": 0, "short": 0, "pseudo": 0, "stand": 0, "taken": 0, "togeth": 0, "session": 0, "equip": 0, "let": [0, 9], "inde": 0, "function": [0, 9], "smi": 0, "wed": 0, "sep": 0, "34": 0, "525": 0, "125": 0, "driver": 0, "12": [0, 8], "persist": 0, "bu": 0, "disp": 0, "volatil": 0, "uncorr": 0, "ecc": 0, "fan": 0, "temp": 0, "perf": 0, "pwr": 0, "cap": 0, "mig": 0, "geforc": 0, "off": 0, "00000000": 0, "41": 0, "42c": 0, "p8": 0, "22w": 0, "240w": 0, "1mib": 0, "8192mib": 0, "gi": 0, "ci": 0, "pid": 0, "No": [0, 8], "found": [0, 6, 12], "look": [0, 7, 9, 10], "abl": [0, 6, 7, 9], "temperatur": 0, "messag": [0, 8], "instead": [0, 9], "though": [0, 2], "contact": [0, 3, 4], "scientif": 0, "team": [0, 3, 4], "requir": [0, 4, 8], "essenti": 0, "think": [0, 4], "central": [0, 3], "environ": [0, 8, 9, 12], "particular": [0, 3, 4, 7, 9], "app": 0, "ubuntu": [0, 10, 12], "final": 0, "mostli": 0, "just": [0, 6, 7, 9], "take": [0, 9], "minut": [0, 8], "subsequ": 0, "tensorflow": 0, "numpi": 0, "21": [0, 9], "6": 0, "109": 0, "x86_64": 0, "debian": 0, "bullsey": 0, "sid": 0, "system_summari": 0, "devic": 0, "physical_devic": 0, "true": 0, "fals": 0, "growth": 0, "none": 0, "tf": 0, "list_physical_devic": 0, "physicaldevic": 0, "device_typ": 0, "constant": 0, "tensor": 0, "shape": 0, "dtype": 0, "string": 0, "b": 0, "encount": 0, "niko": 0, "sirmpilatz": 0, "unit": [0, 4, 6], "logout": [0, 9], "twice": 0, "relat": [1, 2, 5, 12], "neuroscientif": [1, 2], "span": 1, "wide": 1, "rang": 1, "electrophysiologi": [1, 3], "behaviour": [1, 3], "calcium": 1, "imag": [1, 3], "histologi": 1, "The": [1, 2, 4, 7, 8, 9, 10], "focu": 1, "us": [1, 2, 3, 4, 6, 7, 9, 11, 12], "softwar": [1, 2, 3, 5, 11, 12], "task": [1, 8], "concept": 1, "sleap": 1, "modul": [1, 8], "hpc": [1, 7, 8, 12], "cluster": [1, 8, 12], "websit": 2, "form": [2, 3, 5, 12], "well": [2, 3, 4, 9], "collect": [2, 5], "tip": [2, 12], "trick": [2, 12], "analysi": 2, "develop": [2, 4, 12], "neurosci": [2, 4], "maintain": 2, "member": 2, "primarili": 2, "aim": 2, "research": [2, 3, 4], "swc": [2, 3, 4, 8, 10, 12], "gatsbi": [2, 4], "gcnu": [2, 4], "neuroscientist": 2, "organis": [2, 3, 4], "three": [2, 3], "approach": 2, "analys": [2, 3], "program": 2, "issu": [2, 12], "scienc": [2, 4], "effect": 2, "share": [2, 5, 9], "your": [2, 3, 6, 8, 9, 10], "kei": 3, "maxim": 3, "increas": [3, 8], "pressur": 3, "funder": [3, 4], "publish": [3, 4], "alongsid": 3, "public": [3, 9], "our": [3, 9], "maximis": 3, "materi": 3, "few": 3, "restrict": 3, "possibl": [3, 11], "As": 3, "minimum": 3, "underpin": 3, "paper": 3, "made": 3, "origin": [3, 4, 6, 11], "dataset": 3, "replic": 3, "cell": 3, "compos": 3, "datatyp": 3, "must": [3, 4, 9], "deposit": 3, "accept": [3, 9], "either": [3, 4, 9], "mint": 3, "doi": 3, "supplement": 3, "doe": [3, 9, 11], "ideal": 3, "could": [3, 6], "easili": [3, 9], "reproduc": 3, "figur": [3, 9], "build": 3, "those": [3, 7, 9], "host": [3, 9, 10], "github": [3, 4, 6], "gitlab": 3, "individu": 3, "lab": [3, 7], "collabor": [3, 4], "appropri": 3, "licens": [3, 5], "perman": [3, 7], "archiv": 3, "associ": 3, "zenodo": 3, "mani": [3, 4, 8], "come": 3, "categori": 3, "modal": 3, "prefer": 3, "20gb": 3, "exist": [3, 4, 6, 9], "figshar": 3, "easi": 3, "support": 3, "larger": [3, 4], "gin": 3, "system": [3, 9], "particularli": 3, "suit": 3, "ongo": 3, "field": [3, 4, 6], "would": [3, 7, 9], "best": [3, 4], "mandat": 3, "brain": 3, "librari": 3, "larg": 3, "dandi": 3, "optophysiologi": 3, "seri": 3, "nwb": 3, "bioimag": 3, "biolog": 3, "citat": 3, "allow": [3, 4, 7, 8, 9], "adopt": 3, "neurodata": 3, "border": 3, "altern": 3, "standardis": 3, "wish": 3, "neuroblueprint": 3, "metadata": 3, "raw": 3, "technic": 3, "question": [3, 6], "adam": 3, "tyson": 3, "url": [3, 6], "globu": 3, "anyth": [3, 9], "http": [3, 6], "rdr": 3, "the_sainsbury_wellcome_centr": 3, "org": 3, "dryad": 3, "datadryad": 3, "german": 3, "ebrain": 3, "eu": 3, "servic": 3, "sourc": [3, 4], "www": 3, "v2": 3, "opensourcebrain": 3, "brainimagelibrari": 3, "idr": 3, "openmicroscopi": 3, "ebi": 3, "dandiarch": 3, "immunostain": 3, "experi": 3, "nemo": 3, "nemoarch": 3, "omic": 3, "openneuro": 3, "bid": 3, "compliant": 3, "mri": 3, "pet": 3, "eeg": 3, "crcn": 3, "brainglob": 3, "atlas": 3, "neuromorpho": 3, "neuron": 3, "morphologi": 3, "ccdb": 3, "ucsd": 3, "edu": 3, "home": [3, 7, 9, 10], "modeldb": 3, "senselab": 3, "med": 3, "yale": 3, "model": 3, "lawyer": 4, "repres": 4, "legal": 4, "advic": 4, "univers": 4, "colleg": 4, "london": 4, "guidanc": 4, "intellectu": 4, "properti": 4, "rather": 4, "practic": 4, "consult": 4, "busi": 4, "commerci": 4, "potenti": 4, "unless": [4, 8], "claus": 4, "bsd": 4, "openli": 4, "simpli": 4, "spirit": 4, "nobodi": 4, "outsid": 4, "employ": 4, "explicitli": [4, 9], "what": [4, 8], "cannot": 4, "typic": 4, "txt": 4, "ip": [4, 10], "law": 4, "complex": 4, "oss": 4, "reflect": 4, "pick": 4, "choosealicens": 4, "tldrlegal": 4, "often": [4, 8], "tempt": 4, "perhap": 4, "academ": 4, "establish": 4, "understand": 4, "seek": 4, "yourself": [4, 9], "decis": 4, "group": [4, 7], "leader": 4, "depart": 4, "head": 4, "obtain": 4, "contributor": 4, "institut": 4, "accord": 4, "extern": [4, 9], "involv": 4, "tricki": 4, "avoid": [4, 8], "commenc": 4, "tech": 4, "formal": 4, "polici": 4, "check": [4, 6, 7, 9], "consid": 4, "easier": [4, 6, 8, 9], "contribut": 4, "ecosystem": 4, "compat": 4, "otherwis": 4, "virtual": [4, 11], "unrestrict": 4, "liabil": 4, "web": 4, "produc": [4, 8], "button": [4, 9], "templat": [4, 6], "earlier": 4, "still": [4, 7], "under": [4, 8], "term": 4, "carefulli": 4, "outset": 4, "ve": 4, "bug": 4, "read": [4, 7, 9], "instal": [4, 6, 7, 9], "bundl": 4, "third": 4, "parti": 4, "against": 4, "famili": 4, "lgpl": 4, "agpl": 4, "vari": 4, "firstli": 4, "adher": 4, "good": [4, 9], "adapt": 4, "excel": 4, "imperi": 4, "topic": 5, "ref": 6, "doc": 6, "repo": 6, "pip": 6, "At": 6, "ask": [6, 10], "enter": [6, 9], "answer": 6, "did": 6, "later": 6, "github_repository_url": 6, "why": 6, "hash": 6, "older": 6, "had": [6, 9], "search": 6, "histori": 6, "9daec12bffbc32da6a252605c5e67c90028fefc0": 6, "compar": 6, "cookicutt": 6, "respons": [6, 8, 11], "run": [6, 7, 8, 9, 11], "between": [6, 9], "subset": 6, "confirm": 6, "delet": 6, "ad": [6, 7], "creation": 6, "care": 6, "doesn": [6, 9], "remov": [6, 7], "your_fil": 6, "rej": 6, "conflict": 6, "don": [6, 9], "forget": [6, 9], "re": [6, 8, 9, 11], "bit": [6, 7], "tediou": 6, "frequent": 6, "notic": 6, "give": 6, "boolean": 6, "branch": 6, "pr": 6, "main": 6, "action": 6, "fine": 6, "fix": 6, "procedur": 7, "belong": 7, "04": [7, 8], "lt": 7, "administr": 7, "sudo": [7, 11], "local": [7, 9, 10], "cif": 7, "apt": [7, 11], "smb_swc": 7, "plain": 7, "constitut": 7, "secur": [7, 9], "risk": 7, "someon": 7, "access": [7, 8, 9], "harden": 7, "600": 7, "writabl": 7, "owner": 7, "sensibl": 7, "place": 7, "mnt": 7, "media": 7, "boot": 7, "backup": 7, "cp": 7, "bak": 7, "favourit": 7, "end": [7, 8], "gw02": 7, "uid": 7, "1002": 7, "gid": 7, "desir": 7, "previou": 7, "typo": 7, "df": 7, "reboot": 7, "umount": 7, "whatev": 7, "again": [7, 9], "mpi": 8, "pass": 8, "alloc": 8, "These": [8, 9], "my_job": 8, "index": 8, "brief": 8, "demysifi": 8, "descript": 8, "alia": 8, "sinfo": 8, "medium": 8, "identifi": 8, "especi": [8, 9], "training_run_24": 8, "realli": 8, "know": [8, 9], "parallelis": 8, "ntask": 8, "16": 8, "total": 8, "amount": 8, "ram": 8, "8g": 8, "32g": 8, "maximum": 8, "dai": [8, 9], "exce": 8, "hand": 8, "delai": 8, "rtx2080": 8, "rtx5000": 8, "a100_2g": 8, "10gb": 8, "myawesomejob": 8, "debug": 8, "condit": 8, "notifi": 8, "begin": 8, "sent": 8, "integ": 8, "order": 8, "slurm_array_task_id": 8, "variabl": 8, "100": 8, "them": [8, 9], "parallel": 8, "given": 8, "syntax": [8, 9], "prevent": 8, "overload": 8, "socket": 9, "protocol": 9, "integr": 9, "page": 9, "client": 9, "usual": [9, 10], "pre": 9, "note": 9, "proceed": 9, "proce": 9, "emul": 9, "present": 9, "newer": 9, "except": 9, "nativ": 9, "cmd": 9, "browser": 9, "notepad": 9, "edit": 9, "bewar": 9, "visibl": 9, "box": 9, "bastion": 9, "alias": 9, "screen": 9, "stop": 9, "life": 9, "trust": 9, "u": 9, "worth": 9, "effort": 9, "benefit": 9, "studio": 9, "pc": 9, "laptop": 9, "distinguish": 9, "jump": [9, 10], "serv": 9, "entri": 9, "funnel": 9, "through": [9, 11], "control": [9, 12], "reduc": 9, "attack": 9, "surfac": 9, "littl": 9, "power": 9, "job": 9, "shouldn": 9, "els": 9, "light": 9, "computation": 9, "enc1": 9, "node10": 9, "machinc": 9, "actual": 9, "filesystem": 9, "mount": [9, 12], "reach": 9, "layer": 9, "outo": 9, "return": 9, "over": 9, "live": 9, "authorized_kei": 9, "known_host": 9, "yet": 9, "intermedi": 9, "aka": 9, "hostnam": [9, 10], "proxyjump": [9, 10], "directli": 9, "bore": 9, "everi": 9, "authent": 9, "upfront": 9, "ton": 9, "plu": 9, "pair": 9, "privat": 9, "try": 9, "signatur": 9, "server": 9, "valid": 9, "grant": 9, "sever": 9, "cryptograph": 9, "algorithm": 9, "keygen": 9, "ed25519": 9, "strike": 9, "balanc": 9, "speed": 9, "id_ed25519": 9, "choos": 9, "passphras": 9, "extra": 9, "leav": 9, "blank": 9, "pub": 9, "never": 9, "anyon": 9, "id_rsa": 9, "non": 9, "identityfil": 9, "custom": 9, "my": 9, "special": 9, "previous": 9, "target": 9, "congrat": 9, "One": 9, "Then": 9, "left": 9, "corner": 9, "pycharm": 9, "offer": 9, "usecas": 10, "mac": 10, "offic": 10, "append": 10, "serveraliveinterv": 10, "60": 10, "swcuserid": 10, "remotemachineusernam": 10, "172": 10, "243": 10, "000": 10, "address": 10, "got": 10, "cogwheel": 10, "wire": 10, "ipv4": 10, "touch": 10, "vpn": 10, "problem": 11, "solut": 11, "msg": 11, "Not": 11, "dist": 11, "upgrad": 11, "might": 11, "mismatch": 11, "vscode": [11, 12], "vt": 11, "alt": 11, "f3": 11, "gnome": 11, "ternim": 11, "throw": 11, "ye": 11, "usr": 11, "python3": 11, "f2": 11, "chatbot": 11, "wait": 11, "warrant": 12, "troubleshoot": 12, "unmanag": 12, "updat": 12, "cookiecutt": 12, "cruft": 12}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"us": [0, 8, 10], "sleap": 0, "modul": 0, "hpc": [0, 9], "cluster": [0, 9], "abbrevi": [0, 8, 9], "prerequisit": [0, 7, 9], "access": 0, "instal": [0, 11], "your": [0, 4, 7], "local": 0, "pc": 0, "laptop": 0, "mount": [0, 7], "swc": [0, 7, 9], "filesystem": 0, "model": [0, 4], "train": 0, "prepar": 0, "job": [0, 8], "run": 0, "evalu": 0, "infer": 0, "The": 0, "cycl": 0, "troubleshoot": [0, 11], "problem": 0, "data": [1, 3], "analysi": 1, "neuroinformat": 2, "unit": 2, "howto": 2, "main": 2, "section": 2, "share": 3, "background": 3, "requir": 3, "code": [3, 4], "repositori": [3, 4], "structur": 3, "assist": 3, "refer": 3, "potenti": 3, "neurosci": 3, "softwar": 4, "licens": 4, "disclaim": 4, "summari": 4, "why": [4, 9], "choos": 4, "To": 4, "ad": 4, "frequent": 4, "ask": 4, "question": 4, "can": 4, "i": [4, 11], "chang": 4, "later": 4, "how": 4, "do": [4, 9], "know": 4, "when": 4, "my": 4, "readi": 4, "releas": 4, "If": 4, "depend": 4, "anoth": 4, "librari": 4, "more": 4, "restrict": 4, "e": 4, "g": 4, "gpl": 4, "doe": 4, "thi": 4, "affect": 4, "have": 4, "includ": 4, "from": 4, "other": 4, "within": 4, "dl": 4, "acknowledg": 4, "open": [5, 11], "scienc": 5, "updat": [6, 11], "cookiecutt": 6, "gener": [6, 8], "file": [6, 7, 8, 9], "cruft": 6, "setup": 6, "link": 6, "appli": 6, "diff": 6, "commit": 6, "ceph": 7, "storag": 7, "ubuntu": [7, 11], "store": 7, "credenti": 7, "creat": 7, "point": 7, "edit": 7, "fstab": 7, "unmount": 7, "slurm": 8, "argument": 8, "primer": 8, "overview": 8, "commonli": 8, "partit": 8, "queue": 8, "name": 8, "number": 8, "node": 8, "core": 8, "memori": 8, "pool": 8, "all": 8, "time": 8, "limit": 8, "resourc": 8, "gpu": 8, "standard": 8, "output": 8, "error": [8, 11], "email": 8, "notif": 8, "address": 8, "arrai": 8, "set": 9, "up": 9, "ssh": [9, 10], "log": 9, "we": 9, "twice": 9, "config": 9, "kei": 9, "remot": 9, "develop": 9, "an": 10, "unmanag": 10, "machin": 10, "vscode": 10, "distribut": 11, "termin": 11, "after": 11, "new": 11, "python": 11, "version": 11, "detach": 11, "fork": 11, "repo": 11, "github": 11, "program": 12}, "envversion": {"sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.viewcode": 1, "sphinx.ext.intersphinx": 1, "nbsphinx": 4, "sphinx": 60}, "alltitles": {"Use the SLEAP module on the HPC cluster": [[0, "use-the-sleap-module-on-the-hpc-cluster"]], "Abbreviations": [[0, "abbreviations"], [8, "abbreviations"], [9, "abbreviations"]], "Prerequisites": [[0, "prerequisites"], [7, "prerequisites"], [9, "prerequisites"]], "Access to the HPC cluster": [[0, "access-to-the-hpc-cluster"]], "Access to the SLEAP module": [[0, "access-to-the-sleap-module"]], "Install SLEAP on your local PC/laptop": [[0, "install-sleap-on-your-local-pc-laptop"]], "Mount the SWC filesystem on your local PC/laptop": [[0, "mount-the-swc-filesystem-on-your-local-pc-laptop"]], "Model training": [[0, "model-training"]], "Prepare the training job": [[0, "prepare-the-training-job"]], "Run the training job": [[0, "run-the-training-job"]], "Evaluate the trained models": [[0, "evaluate-the-trained-models"]], "Model inference": [[0, "model-inference"]], "The training-inference cycle": [[0, "the-training-inference-cycle"]], "Troubleshooting": [[0, "troubleshooting"], [11, "troubleshooting"]], "Problems with the SLEAP module": [[0, "problems-with-the-sleap-module"]], "Data Analysis": [[1, "data-analysis"]], "Neuroinformatics Unit HowTo": [[2, "neuroinformatics-unit-howto"]], "Main sections": [[2, "main-sections"]], "Data Sharing": [[3, "data-sharing"]], "Background": [[3, "background"]], "Requirements": [[3, "requirements"]], "Code": [[3, "code"]], "Data": [[3, "data"]], "Repository": [[3, "repository"]], "Structure": [[3, "structure"]], "Assistance": [[3, "assistance"]], "Reference of potential repositories for neuroscience data": [[3, "reference-of-potential-repositories-for-neuroscience-data"]], "Software licensing": [[4, "software-licensing"]], "Disclaimer": [[4, "disclaimer"]], "Summary": [[4, "summary"]], "Why license your software?": [[4, "why-license-your-software"]], "Choosing a license": [[4, "choosing-a-license"]], "To choose a license:": [[4, "to-choose-a-license"]], "Adding a license": [[4, "adding-a-license"]], "Frequently asked questions": [[4, "frequently-asked-questions"]], "Can I change the license later?": [[4, "can-i-change-the-license-later"]], "How do I know when my code is ready to release?": [[4, "how-do-i-know-when-my-code-is-ready-to-release"]], "If my code depends on another library with a more restrictive license (e.g. GPL), does this affect how I license my software?": [[4, "if-my-code-depends-on-another-library-with-a-more-restrictive-license-e-g-gpl-does-this-affect-how-i-license-my-software"]], "I have included code from others within my repository (e.g. a DL model), how does this affect licensing?": [[4, "i-have-included-code-from-others-within-my-repository-e-g-a-dl-model-how-does-this-affect-licensing"]], "Acknowledgements": [[4, "acknowledgements"]], "Open Science": [[5, "open-science"]], "Update cookiecutter-generated files with cruft": [[6, "update-cookiecutter-generated-files-with-cruft"]], "Setup: cruft link": [[6, "setup-cruft-link"]], "Apply updates: cruft diff & cruft update": [[6, "apply-updates-cruft-diff-cruft-update"]], "Commit": [[6, "commit"]], "Mount ceph storage on Ubuntu": [[7, "mount-ceph-storage-on-ubuntu"]], "Store your SWC credentials": [[7, "store-your-swc-credentials"]], "Create a mount point": [[7, "create-a-mount-point"]], "Edit the fstab file": [[7, "edit-the-fstab-file"]], "Mount the storage": [[7, "mount-the-storage"]], "Unmount the storage": [[7, "unmount-the-storage"]], "SLURM arguments primer": [[8, "slurm-arguments-primer"]], "Overview": [[8, "overview"]], "Commonly used arguments": [[8, "commonly-used-arguments"]], "Partition (Queue)": [[8, "partition-queue"]], "Job Name": [[8, "job-name"]], "Number of Nodes": [[8, "number-of-nodes"]], "Number of Cores": [[8, "number-of-cores"]], "Memory Pool for All Cores": [[8, "memory-pool-for-all-cores"]], "Time Limit": [[8, "time-limit"]], "Generic Resources (GPUs)": [[8, "generic-resources-gpus"]], "Standard Output File": [[8, "standard-output-file"]], "Standard Error File": [[8, "standard-error-file"]], "Email Notifications": [[8, "email-notifications"]], "Email Address": [[8, "email-address"]], "Array jobs": [[8, "array-jobs"]], "Set up SSH for the SWC HPC cluster": [[9, "set-up-ssh-for-the-swc-hpc-cluster"]], "Log into the cluster": [[9, "log-into-the-cluster"]], "Why do we SSH twice?": [[9, "why-do-we-ssh-twice"]], "SSH config file": [[9, "ssh-config-file"]], "SSH keys": [[9, "ssh-keys"]], "Remote development": [[9, "remote-development"]], "SSH into an unmanaged machine using VSCode": [[10, "ssh-into-an-unmanaged-machine-using-vscode"]], "Ubuntu distribution update error": [[11, "ubuntu-distribution-update-error"]], "Terminal is not opening after installing a new Python version": [[11, "terminal-is-not-opening-after-installing-a-new-python-version"]], "Detach a forked repo on GitHub": [[11, "detach-a-forked-repo-on-github"]], "Programming": [[12, "programming"]]}, "indexentries": {}}) \ No newline at end of file +Search.setIndex({"docnames": ["data_analysis/index", "index", "open_science/Data-sharing", "open_science/Licensing", "open_science/index", "programming/Cookiecutter-cruft", "programming/Mount-ceph-ubuntu", "programming/Mount-ceph-ubuntu-temp", "programming/SSH-SWC-cluster", "programming/SSH-vscode", "programming/Troubleshooting", "programming/index"], "filenames": ["data_analysis/index.md", "index.md", "open_science/Data-sharing.md", "open_science/Licensing.md", "open_science/index.md", "programming/Cookiecutter-cruft.md", "programming/Mount-ceph-ubuntu.md", "programming/Mount-ceph-ubuntu-temp.md", "programming/SSH-SWC-cluster.md", "programming/SSH-vscode.md", "programming/Troubleshooting.md", "programming/index.md"], "titles": ["Data Analysis", "Neuroinformatics Unit HowTo", "Data Sharing", "Software licensing", "Open Science", "Update cookiecutter-generated files with cruft", "Mount ceph storage on Ubuntu permanently", "Mount ceph storage on Ubuntu temporarily", "Set up SSH for the SWC HPC cluster", "SSH into an unmanaged machine using VSCode", "Troubleshooting", "Programming"], "terms": {"guid": [0, 1, 3, 4, 8, 11], "relat": [0, 1, 4, 11], "neuroscientif": [0, 1], "span": 0, "wide": 0, "rang": 0, "type": [0, 2, 8], "electrophysiologi": [0, 2], "behaviour": [0, 2], "calcium": 0, "imag": [0, 2], "histologi": 0, "etc": [0, 2, 6, 8, 11], "The": [0, 1, 3, 6, 7, 8, 9], "focu": 0, "mai": [0, 2, 3, 8], "us": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11], "specif": [0, 2, 3], "softwar": [0, 1, 2, 4, 10, 11], "tool": [0, 1, 3, 8, 9], "more": [0, 5, 8], "gener": [0, 1, 2, 8, 11], "task": 0, "concept": 0, "On": [1, 8, 9], "thi": [1, 2, 5, 6, 7, 8, 9, 10], "websit": 1, "you": [1, 2, 3, 5, 6, 7, 8, 9, 10], "can": [1, 2, 5, 6, 7, 8, 9, 10, 11], "find": [1, 3, 6, 8, 9], "long": [1, 4, 8, 11], "form": [1, 2, 4, 11], "how": [1, 7, 8], "well": [1, 2, 3, 8], "collect": [1, 4], "small": [1, 2, 10, 11], "tip": [1, 11], "trick": [1, 11], "data": [1, 4], "analysi": 1, "develop": [1, 3, 11], "neurosci": [1, 3], "i": [1, 2, 5, 6, 8, 9], "maintain": 1, "member": 1, "inform": [1, 2, 3, 7], "primarili": 1, "aim": 1, "research": [1, 2, 3], "sainsburi": [1, 3, 8], "wellcom": [1, 2, 3, 8], "centr": [1, 3, 8], "swc": [1, 2, 3, 7, 9, 11], "gatsbi": [1, 3], "comput": [1, 2, 3, 8], "gcnu": [1, 3], "though": 1, "most": [1, 3, 8], "should": [1, 2, 3, 5, 6, 7, 8], "neuroscientist": 1, "ar": [1, 2, 3, 5, 6, 8, 9], "organis": [1, 2, 3], "three": [1, 2], "approach": 1, "analys": [1, 2], "program": 1, "code": [1, 8, 11], "issu": [1, 11], "open": [1, 2, 3, 5, 6, 8, 9], "scienc": [1, 3], "effect": 1, "share": [1, 4, 7, 8], "your": [1, 2, 5, 7, 8, 9], "kei": 2, "maxim": 2, "util": [2, 6, 7, 8], "There": [2, 3, 8], "increas": 2, "pressur": 2, "from": [2, 5, 8, 9, 10], "funder": [2, 3], "publish": [2, 3], "alongsid": 2, "public": [2, 8], "we": [2, 3, 5, 6, 7], "expect": [2, 8], "our": [2, 7, 8], "maximis": 2, "avail": [2, 8], "materi": 2, "few": 2, "restrict": 2, "possibl": [2, 10], "As": 2, "minimum": 2, "underpin": 2, "paper": 2, "made": 2, "other": [2, 6, 8], "time": [2, 3, 5, 6, 8], "ani": [2, 3, 8], "origin": [2, 3, 5, 10], "view": [2, 8], "dataset": 2, "replic": 2, "cell": 2, "press": [2, 5, 8], "all": [2, 5, 8, 10], "compos": 2, "standard": 2, "datatyp": 2, "report": 2, "must": [2, 3, 8], "deposit": 2, "recommend": [2, 3, 8], "befor": [2, 3, 5, 6, 8], "accept": [2, 8], "either": [2, 3, 8], "mint": 2, "doi": 2, "includ": [2, 8], "supplement": 2, "current": [2, 10], "doe": [2, 7, 8, 10], "have": [2, 5, 6, 8, 9], "do": [2, 5, 6, 7, 9, 11], "ideal": 2, "wai": [2, 3, 9], "could": [2, 5], "easili": [2, 8], "reproduc": 2, "figur": [2, 8], "build": 2, "those": [2, 6, 8], "own": [2, 3], "host": [2, 8, 9], "github": [2, 3, 5], "gitlab": 2, "an": [2, 3, 6, 7, 8, 10, 11], "account": [2, 8], "e": [2, 5, 6, 7, 8], "individu": 2, "particular": [2, 3, 6, 7, 8], "team": [2, 3], "lab": [2, 6, 7], "collabor": [2, 3], "To": [2, 6, 8], "enabl": 2, "reus": 2, "appropri": [2, 7], "licens": [2, 4], "ensur": [2, 3, 6], "perman": [2, 7, 11], "archiv": 2, "associ": 2, "creat": [2, 5, 8, 9], "g": [2, 6, 7, 8], "zenodo": 2, "mani": [2, 3], "thei": [2, 3, 5, 6, 8], "come": 2, "categori": 2, "modal": 2, "If": [2, 5, 6, 7, 8, 9, 10], "prefer": 2, "For": [2, 3, 5, 6, 8], "20gb": 2, "exist": [2, 3, 5, 8], "section": [2, 8, 11], "ucl": [2, 3, 6, 7, 8, 9], "figshar": 2, "instanc": 2, "easi": 2, "support": 2, "central": 2, "larger": [2, 3], "gin": 2, "within": [2, 8], "git": [2, 5, 8], "provid": [2, 5, 8], "version": [2, 3, 5, 11], "system": [2, 7, 8], "make": [2, 5, 6, 7, 8, 9], "particularli": 2, "suit": 2, "ongo": 2, "project": [2, 6], "In": [2, 3, 5, 6, 7, 8, 9], "some": [2, 8], "case": [2, 3, 5, 6, 7, 8], "field": [2, 3, 5], "would": [2, 6, 7, 8], "best": [2, 3], "mandat": 2, "exampl": [2, 5, 6, 7, 8, 9], "brain": 2, "librari": 2, "larg": 2, "dandi": 2, "optophysiologi": 2, "seri": 2, "nwb": 2, "format": [2, 6], "bioimag": 2, "biolog": 2, "help": [2, 3, 7], "citat": 2, "singl": [2, 8], "allow": [2, 3, 6, 8], "user": [2, 6, 7, 8, 9], "purpos": 2, "adopt": 2, "file": [2, 3, 7, 9, 10, 11], "neurodata": 2, "without": [2, 3, 7, 8, 10], "border": 2, "altern": 2, "standardis": 2, "folder": [2, 8, 9], "start": [2, 8], "scratch": 2, "wish": 2, "neuroblueprint": 2, "howev": [2, 3, 6, 8], "relev": 2, "metadata": 2, "raw": 2, "technic": 2, "question": [2, 5], "about": [2, 3, 5, 7, 8], "pleas": [2, 3, 5], "contact": [2, 3], "adam": 2, "tyson": 2, "ac": [2, 6, 7, 8, 9], "uk": [2, 6, 7, 8, 9], "url": [2, 5], "domain": [2, 6, 7], "globu": 2, "anyth": [2, 8], "http": [2, 5], "rdr": 2, "the_sainsbury_wellcome_centr": 2, "org": 2, "dryad": 2, "datadryad": 2, "german": 2, "neuroinformat": [2, 3, 5, 6, 7], "node": [2, 8], "ebrain": 2, "eu": 2, "servic": 2, "sourc": [2, 3], "www": 2, "v2": 2, "opensourcebrain": 2, "elsewher": 2, "brainimagelibrari": 2, "resourc": [2, 3, 8], "idr": 2, "openmicroscopi": 2, "ebi": 2, "dandiarch": 2, "immunostain": 2, "experi": 2, "nemo": 2, "nemoarch": 2, "omic": 2, "initi": 2, "openneuro": 2, "bid": 2, "compliant": 2, "mri": 2, "pet": 2, "eeg": 2, "crcn": 2, "brainglob": 2, "atlas": 2, "neuromorpho": 2, "neuron": 2, "morphologi": 2, "ccdb": 2, "ucsd": 2, "edu": 2, "home": [2, 6, 8, 9], "modeldb": 2, "senselab": 2, "med": 2, "yale": 2, "model": 2, "document": [3, 8], "wa": 3, "written": 3, "lawyer": 3, "repres": 3, "legal": 3, "advic": 3, "unit": [3, 5], "univers": 3, "colleg": 3, "london": 3, "It": [3, 5, 7, 8], "offici": 3, "guidanc": 3, "intellectu": 3, "properti": 3, "rather": 3, "practic": 3, "consult": 3, "busi": 3, "think": 3, "commerci": 3, "potenti": 3, "unless": 3, "ha": [3, 5, 8], "upon": [3, 7], "3": [3, 10], "claus": 3, "bsd": 3, "reason": 3, "openli": 3, "requir": 3, "simpli": 3, "spirit": 3, "nobodi": 3, "outsid": 3, "employ": 3, "": [3, 5, 8, 10], "explicitli": [3, 8], "detail": 3, "what": 3, "cannot": 3, "typic": 3, "txt": 3, "ip": [3, 9], "law": 3, "complex": 3, "oss": 3, "select": [3, 8], "reflect": 3, "pick": 3, "choosealicens": 3, "com": [3, 5], "tldrlegal": 3, "often": 3, "tempt": 3, "write": 3, "perhap": 3, "onli": [3, 5, 6, 7, 8], "academ": 3, "one": [3, 8, 9], "establish": 3, "understand": 3, "seek": 3, "permiss": [3, 6], "yourself": [3, 8], "decis": 3, "up": [3, 5, 7, 11], "group": [3, 6], "leader": 3, "depart": 3, "head": 3, "obtain": 3, "contributor": 3, "part": 3, "differ": [3, 8], "institut": 3, "accord": 3, "extern": [3, 8], "involv": 3, "tricki": 3, "avoid": 3, "commenc": 3, "work": [3, 8, 9], "tech": 3, "transfer": 3, "follow": [3, 6, 7, 8, 9, 10], "which": [3, 6, 8], "formal": 3, "polici": 3, "check": [3, 5, 6, 8], "consid": 3, "easier": [3, 5, 8], "contribut": 3, "ecosystem": 3, "compat": 3, "otherwis": 3, "want": [3, 5, 8, 10], "simpl": [3, 8], "virtual": [3, 10], "unrestrict": 3, "liabil": 3, "copi": [3, 8], "text": [3, 6, 8], "name": [3, 6, 7, 8, 10], "similar": [3, 8], "new": [3, 5, 6, 8], "web": 3, "interfac": [3, 8], "produc": 3, "button": [3, 8], "templat": [3, 5], "list": [3, 7, 8], "earlier": 3, "still": [3, 6], "under": [3, 7], "term": 3, "import": 3, "carefulli": 3, "outset": 3, "ve": 3, "abov": [3, 6, 8], "step": [3, 8], "alwai": 3, "bug": 3, "read": [3, 6, 8], "instal": [3, 5, 6, 7, 8], "separ": 3, "call": 3, "python": [3, 5, 8], "packag": [3, 5], "normal": 3, "bundl": 3, "third": 3, "parti": 3, "link": [3, 8], "against": 3, "famili": 3, "lgpl": 3, "agpl": 3, "vari": 3, "firstli": 3, "also": [3, 5, 8], "adher": 3, "good": [3, 8], "adapt": 3, "excel": 3, "imperi": 3, "A": [4, 8], "topic": 4, "ref": 5, "doc": 5, "repo": 5, "pip": 5, "first": [5, 6, 8], "set": [5, 7, 9, 11], "command": [5, 6, 7, 8, 11], "specifi": [5, 8], "At": 5, "point": [5, 8], "ask": [5, 9], "same": [5, 6, 8], "were": 5, "when": [5, 7, 8, 9, 10], "enter": [5, 8], "answer": 5, "did": 5, "sure": [5, 6, 7, 9], "later": 5, "prompt": [5, 7, 8], "github_repository_url": 5, "why": 5, "found": [5, 11], "correct": 5, "fail": 5, "after": 5, "hash": 5, "older": 5, "had": [5, 8], "now": [5, 6, 8], "manual": 5, "search": 5, "histori": 5, "9daec12bffbc32da6a252605c5e67c90028fefc0": 5, "need": [5, 8], "becaus": [5, 7, 8], "compar": 5, "chang": [5, 6, 10], "sinc": [5, 8], "json": 5, "contain": [5, 8], "cookicutt": 5, "respons": [5, 10], "track": 5, "run": [5, 6, 7, 8, 10], "see": [5, 6, 7, 8], "between": [5, 8], "subset": 5, "confirm": 5, "y": 5, "v": [5, 8], "delet": 5, "ad": [5, 6, 7], "its": 5, "creation": 5, "care": 5, "doesn": [5, 8], "t": [5, 7, 8], "been": [5, 6], "remov": [5, 6], "abl": [5, 6, 8], "your_fil": 5, "rej": 5, "each": 5, "conflict": 5, "base": 5, "content": [5, 6, 8], "don": [5, 8], "forget": [5, 8], "re": [5, 8, 10], "done": [5, 6, 8], "bit": [5, 6], "tediou": 5, "get": [5, 6, 7, 8, 10], "frequent": 5, "notic": 5, "futur": 5, "date": 5, "just": [5, 6, 7, 8], "give": 5, "boolean": 5, "onc": [5, 8], "satisfi": 5, "branch": 5, "pr": 5, "merg": 5, "main": 5, "action": 5, "everyth": 5, "fine": 5, "fix": 5, "error": [5, 6, 7], "repeat": 5, "process": [5, 8], "partit": 6, "procedur": 6, "belong": 6, "temporarili": [6, 11], "instruct": [6, 10], "test": 6, "20": 6, "04": 6, "lt": 6, "administr": [6, 7], "right": [6, 7], "sudo": [6, 7, 10], "local": [6, 7, 8, 9], "machin": [6, 7, 8, 11], "cif": [6, 7], "via": [6, 7, 8, 10], "apt": [6, 7, 10], "login": 6, "save": [6, 8], "directori": [6, 7, 8], "termin": [6, 8], "nano": [6, 8, 10], "smb_swc": 6, "look": [6, 8, 9], "like": [6, 8], "usernam": [6, 7, 8], "password": [6, 7, 8, 9], "exit": [6, 8, 10], "plain": 6, "constitut": 6, "secur": [6, 8], "risk": 6, "someon": 6, "access": [6, 8], "harden": 6, "chmod": 6, "600": 6, "readabl": 6, "writabl": 6, "owner": 6, "log": 6, "sensibl": [6, 7], "place": [6, 7], "mnt": [6, 7], "media": [6, 7], "below": [6, 7, 8], "mkdir": [6, 7], "automat": [6, 8], "boot": 6, "backup": 6, "cp": 6, "bak": 6, "favourit": 6, "editor": [6, 8], "add": [6, 8], "line": [6, 8, 11], "end": 6, "gw02": [6, 7], "hpc": [6, 7, 11], "uid": 6, "1002": 6, "gid": 6, "0": 6, "replac": [6, 7, 9], "path": [6, 7, 8], "desir": [6, 7], "previou": [6, 7], "id": [6, 8, 11], "out": [6, 7, 8], "typo": 6, "df": [6, 7], "h": [6, 7], "next": [6, 7, 8, 9], "reboot": [6, 7], "umount": [6, 7], "whatev": 6, "again": [6, 8], "temporari": 7, "remain": 7, "until": 7, "option": [7, 8], "o": [7, 8], "full": 7, "execut": 7, "show": 7, "print": [7, 8], "rememb": 7, "verifi": [7, 8], "output": [7, 8], "network": [7, 8, 9], "drive": 7, "linux": [7, 8, 9, 11], "smb": 7, "explain": 8, "connect": [8, 9], "intern": 8, "wiki": 8, "interpret": 8, "block": 8, "shell": 8, "shown": 8, "sign": 8, "indic": 8, "echo": 8, "hello": 8, "world": 8, "similarli": 8, "appear": [8, 10], "both": 8, "acronym": 8, "mean": 8, "high": 8, "perform": 8, "slurm": 8, "manag": 8, "socket": 8, "protocol": 8, "integr": 8, "environ": [8, 11], "gui": 8, "graphic": 8, "know": 8, "especi": 8, "page": 8, "basic": 8, "navig": 8, "client": 8, "usual": [8, 9], "pre": 8, "maco": 8, "window": [8, 9], "10": [8, 10], "note": 8, "proceed": 8, "two": 8, "proce": 8, "bash": 8, "emul": 8, "default": 8, "along": 8, "present": 8, "assum": 8, "newer": 8, "except": 8, "nativ": 8, "function": 8, "describ": 8, "here": 8, "cmd": 8, "gw1": 8, "browser": 8, "notepad": 8, "instead": 8, "c": 8, "alreadi": 8, "edit": 8, "bewar": 8, "extens": [8, 9], "visibl": 8, "click": [8, 9], "box": 8, "dai": 8, "gatewai": 8, "bastion": 8, "defin": 8, "alias": 8, "displai": 8, "screen": 8, "successfulli": 8, "stop": 8, "learn": 8, "life": 8, "trust": 8, "u": 8, "worth": 8, "effort": 8, "benefit": 8, "visual": 8, "studio": 8, "pc": 8, "laptop": 8, "last": 8, "distinguish": 8, "jump": [8, 9], "serv": 8, "entri": 8, "By": 8, "funnel": 8, "through": [8, 10], "monitor": 8, "control": [8, 11], "reduc": 8, "attack": 8, "surfac": 8, "veri": 8, "littl": 8, "power": 8, "submit": 8, "job": 8, "shouldn": 8, "els": 8, "light": 8, "script": 8, "computation": 8, "intens": 8, "across": 8, "enc1": 8, "node10": 8, "gpu": 8, "sr670": 8, "21": 8, "These": 8, "machinc": 8, "actual": 8, "interact": 8, "srun": 8, "batch": 8, "sbatch": 8, "locat": 8, "where": 8, "filesystem": 8, "ceph": [8, 11], "mount": [8, 11], "take": 8, "second": 8, "reach": 8, "logout": 8, "layer": 8, "outo": 8, "return": 8, "over": 8, "live": 8, "cd": [8, 9], "l": 8, "1": 8, "authorized_kei": 8, "known_host": 8, "yet": 8, "intermedi": 8, "aka": 8, "hostnam": [8, 9], "proxyjump": [8, 9], "ctrl": [8, 10], "x": 8, "directli": 8, "syntax": 8, "bore": 8, "everi": 8, "authent": 8, "upfront": 8, "ton": 8, "plu": 8, "pair": 8, "privat": 8, "try": 8, "signatur": 8, "server": 8, "valid": 8, "grant": 8, "sever": 8, "cryptograph": 8, "algorithm": 8, "argument": 8, "keygen": 8, "ed25519": 8, "strike": 8, "balanc": 8, "speed": 8, "id_ed25519": 8, "choos": 8, "anoth": 8, "passphras": 8, "extra": 8, "leav": 8, "blank": 8, "pub": 8, "never": 8, "anyon": 8, "id_rsa": 8, "non": 8, "multipl": 8, "identityfil": 8, "direct": 8, "custom": 8, "my": 8, "special": 8, "let": 8, "configur": [8, 9], "previous": 8, "target": 8, "them": 8, "That": 8, "congrat": 8, "so": 8, "One": 8, "Then": 8, "bottom": 8, "left": 8, "corner": 8, "pycharm": 8, "offer": 8, "usecas": 9, "mac": 9, "offic": 9, "config": 9, "append": 9, "serveraliveinterv": 9, "60": 9, "swcuserid": 9, "remot": 9, "remotemachineusernam": 9, "172": 9, "24": 9, "243": 9, "000": 9, "address": 9, "ubuntu": [9, 11], "got": 9, "cogwheel": 9, "wire": 9, "ipv4": 9, "touch": 9, "vpn": 9, "problem": 10, "solut": 10, "msg": 10, "Not": 10, "dist": 10, "upgrad": 10, "conda": 10, "might": 10, "mismatch": 10, "bin": 10, "vscode": [10, 11], "vt": 10, "alt": 10, "f3": 10, "gnome": 10, "ternim": 10, "throw": 10, "ye": 10, "usr": 10, "python3": 10, "f2": 10, "chatbot": 10, "assist": 10, "wait": 10, "warrant": 11, "troubleshoot": 11, "ssh": 11, "cluster": 11, "unmanag": 11, "storag": 11, "updat": 11, "cookiecutt": 11, "cruft": 11}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"data": [0, 2], "analysi": 0, "neuroinformat": 1, "unit": 1, "howto": 1, "main": 1, "section": 1, "share": 2, "background": 2, "requir": 2, "code": [2, 3], "repositori": [2, 3], "structur": 2, "assist": 2, "refer": [2, 7], "potenti": 2, "neurosci": 2, "softwar": 3, "licens": 3, "disclaim": 3, "summari": 3, "why": [3, 8], "your": [3, 6], "choos": 3, "To": [3, 7], "ad": 3, "frequent": 3, "ask": 3, "question": 3, "can": 3, "i": [3, 7, 10], "chang": 3, "later": 3, "how": 3, "do": [3, 8], "know": 3, "when": 3, "my": 3, "readi": 3, "releas": 3, "If": 3, "depend": 3, "anoth": 3, "librari": 3, "more": 3, "restrict": 3, "e": 3, "g": 3, "gpl": 3, "doe": 3, "thi": 3, "affect": 3, "have": 3, "includ": 3, "from": 3, "other": 3, "within": 3, "dl": 3, "model": 3, "acknowledg": 3, "open": [4, 10], "scienc": 4, "updat": [5, 10], "cookiecutt": 5, "gener": 5, "file": [5, 6, 8], "cruft": 5, "setup": 5, "link": 5, "appli": 5, "diff": 5, "commit": 5, "mount": [6, 7], "ceph": [6, 7], "storag": [6, 7], "ubuntu": [6, 7, 10], "perman": 6, "prerequisit": [6, 7, 8], "step": [6, 7], "1": [6, 7], "store": 6, "swc": [6, 8], "credenti": 6, "2": [6, 7], "creat": [6, 7], "point": [6, 7], "3": [6, 7], "edit": 6, "fstab": 6, "4": [6, 7], "5": 6, "unmount": [6, 7], "temporarili": 7, "partit": 7, "check": 7, "correctli": 7, "set": 8, "up": 8, "ssh": [8, 9], "hpc": 8, "cluster": 8, "abbrevi": 8, "log": 8, "we": 8, "twice": 8, "config": 8, "kei": 8, "remot": 8, "develop": 8, "an": 9, "unmanag": 9, "machin": 9, "us": 9, "vscode": 9, "troubleshoot": 10, "distribut": 10, "error": 10, "termin": 10, "after": 10, "instal": 10, "new": 10, "python": 10, "version": 10, "detach": 10, "fork": 10, "repo": 10, "github": 10, "program": 11}, "envversion": {"sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.viewcode": 1, "sphinx.ext.intersphinx": 1, "nbsphinx": 4, "sphinx": 60}, "alltitles": {"Data Analysis": [[0, "data-analysis"]], "Neuroinformatics Unit HowTo": [[1, "neuroinformatics-unit-howto"]], "Main sections": [[1, "main-sections"]], "Data Sharing": [[2, "data-sharing"]], "Background": [[2, "background"]], "Requirements": [[2, "requirements"]], "Code": [[2, "code"]], "Data": [[2, "data"]], "Repository": [[2, "repository"]], "Structure": [[2, "structure"]], "Assistance": [[2, "assistance"]], "Reference of potential repositories for neuroscience data": [[2, "reference-of-potential-repositories-for-neuroscience-data"]], "Software licensing": [[3, "software-licensing"]], "Disclaimer": [[3, "disclaimer"]], "Summary": [[3, "summary"]], "Why license your software?": [[3, "why-license-your-software"]], "Choosing a license": [[3, "choosing-a-license"]], "To choose a license:": [[3, "to-choose-a-license"]], "Adding a license": [[3, "adding-a-license"]], "Frequently asked questions": [[3, "frequently-asked-questions"]], "Can I change the license later?": [[3, "can-i-change-the-license-later"]], "How do I know when my code is ready to release?": [[3, "how-do-i-know-when-my-code-is-ready-to-release"]], "If my code depends on another library with a more restrictive license (e.g. GPL), does this affect how I license my software?": [[3, "if-my-code-depends-on-another-library-with-a-more-restrictive-license-e-g-gpl-does-this-affect-how-i-license-my-software"]], "I have included code from others within my repository (e.g. a DL model), how does this affect licensing?": [[3, "i-have-included-code-from-others-within-my-repository-e-g-a-dl-model-how-does-this-affect-licensing"]], "Acknowledgements": [[3, "acknowledgements"]], "Open Science": [[4, "open-science"]], "Update cookiecutter-generated files with cruft": [[5, "update-cookiecutter-generated-files-with-cruft"]], "Setup: cruft link": [[5, "setup-cruft-link"]], "Apply updates: cruft diff & cruft update": [[5, "apply-updates-cruft-diff-cruft-update"]], "Commit": [[5, "commit"]], "Mount ceph storage on Ubuntu permanently": [[6, "mount-ceph-storage-on-ubuntu-permanently"]], "Prerequisites": [[6, "prerequisites"], [7, "prerequisites"], [8, "prerequisites"]], "Steps": [[6, "steps"], [7, "steps"]], "1. Store your SWC credentials": [[6, "store-your-swc-credentials"]], "2. Create a mount point": [[6, "create-a-mount-point"]], "3. Edit the fstab file": [[6, "edit-the-fstab-file"]], "4. Mount the storage": [[6, "mount-the-storage"]], "5. Unmount the storage": [[6, "unmount-the-storage"]], "Mount ceph storage on Ubuntu temporarily": [[7, "mount-ceph-storage-on-ubuntu-temporarily"]], "1. Create a mount point": [[7, "create-a-mount-point"]], "2. Mount the ceph partition": [[7, "mount-the-ceph-partition"]], "3. Check the partition is mounted correctly": [[7, "check-the-partition-is-mounted-correctly"]], "4. To unmount the storage": [[7, "to-unmount-the-storage"]], "References": [[7, "references"]], "Set up SSH for the SWC HPC cluster": [[8, "set-up-ssh-for-the-swc-hpc-cluster"]], "Abbreviations": [[8, "abbreviations"]], "Log into the cluster": [[8, "log-into-the-cluster"]], "Why do we SSH twice?": [[8, "why-do-we-ssh-twice"]], "SSH config file": [[8, "ssh-config-file"]], "SSH keys": [[8, "ssh-keys"]], "Remote development": [[8, "remote-development"]], "SSH into an unmanaged machine using VSCode": [[9, "ssh-into-an-unmanaged-machine-using-vscode"]], "Troubleshooting": [[10, "troubleshooting"]], "Ubuntu distribution update error": [[10, "ubuntu-distribution-update-error"]], "Terminal is not opening after installing a new Python version": [[10, "terminal-is-not-opening-after-installing-a-new-python-version"]], "Detach a forked repo on GitHub": [[10, "detach-a-forked-repo-on-github"]], "Programming": [[11, "programming"]]}, "indexentries": {}}) \ No newline at end of file