diff --git a/content/courses/parallel-computing-introduction/codes/a.out b/content/courses/parallel-computing-introduction/codes/a.out index 7f61305b..b772d8f1 100755 Binary files a/content/courses/parallel-computing-introduction/codes/a.out and b/content/courses/parallel-computing-introduction/codes/a.out differ diff --git a/content/courses/parallel-computing-introduction/codes/manager_worker_stub.cxx b/content/courses/parallel-computing-introduction/codes/manager_worker_stub.cxx new file mode 100644 index 00000000..60b399df --- /dev/null +++ b/content/courses/parallel-computing-introduction/codes/manager_worker_stub.cxx @@ -0,0 +1,77 @@ +#include +#include +#include +#include +#include +#include +#include +#include "mpi.h" + + +using namespace std; + +int random_int(int n,int m) { + //quick and stupid way, using full C++ machinery is better but complex + return n+rand()%m; +} + +double do_work() { + //hardcoded bounds for convenience + int nsteps=random_int(10000,30000); + float result=0.; + for (int i=0; i results(nprocs); + + int done=0; + float result; + int sender; + + if (rank==0) { + for (int i=1; i}} +{{< code-download file="/courses/parallel-computing-introduction/solns/ring.cxx" lang="c++" >}} +{{< /spoiler >}} +{{< spoiler text="Fortran" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/ring.f90" lang="fortran" >}} +{{< /spoiler >}} +{{< spoiler text="Python" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/ring.py" lang="python" >}} +{{< /spoiler >}} + +## Project 5 + +Write a program in which all processes send a message to their left and receive from their right. Your program should handle all process counts appropriately. + +1. Left end sends nothing, right end receives nothing. +2. Make the messages circular, i.e. 0 sends to np-1 and np-1 receives from 0. + +#### Example Solutions +{{< spoiler text="C++" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/shift_1.cxx" lang="c++" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/shift_2.cxx" lang="c++" >}} +{{< /spoiler >}} +{{< spoiler text="Fortran" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/shift_1.f90" lang="fortran" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/shift_2.f90" lang="fortran" >}} +{{< /spoiler >}} +{{< spoiler text="Python" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/shift_1.py" lang="python" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/shift_2.py" lang="python" >}} +{{< /spoiler >}} + +## Project 6 + +Now write a program in which all processes send a message to their left and receive from their right, then send a different message to the right and receive from the left. Your program should handle all process counts appropriately. + +1. Left end sends only to the right, right end receives only from the left. +2. Make the messages circular, i.e. 0 sends to np-1 and np-1 receives from 0. + +In this case, watch out for deadlock or unsafe patterns. + + +## Project 7 + +A common pattern in parallel programming is the _manager-worker_ structure. A "manager" process distributes work to "worker" processes. Sometimes manager code is separated by `if rank==0` (the manager should nearly always be rank 0) statements, while the other ranks execute the "worker" code. Sometimes the manager spawns distinct worker processes, but that requires using the more advanced MPI `spawn` capability. In this project we will use a single code. Usually the manager distributes work to the workers, which return results; the manager then hands more work to those processes that are ready, until all is completed. For this example the workers will do only one round of work. + +Starting from the stub for your language, complete the send and receive calls. + +#### Starting Codes +{{< spoiler text="C++" >}} +{{< code-download file="/courses/parallel-computing-introduction/codes/manager_worker_stub.cxx" lang="c++" >}} +{{< /spoiler >}} +{{< spoiler text="Fortran" >}} +{{< code-download file="/courses/parallel-computing-introduction/codes/manager_worker_stub.f90" lang="fortran" >}} +{{< /spoiler >}} +{{< spoiler text="Python" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/manager_worker.py" lang="python" >}} +{{< /spoiler >}} + +#### Example Solutions +{{< spoiler text="C++" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/manager_worker.cxx" lang="c++" >}} +{{< /spoiler >}} +{{< spoiler text="Fortran" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/manager_worker.f90" lang="fortran" >}} +{{< /spoiler >}} +{{< spoiler text="Python" >}} +{{< code-download file="/courses/parallel-computing-introduction/solns/manager_worker.py" lang="python" >}} +{{< /spoiler >}} diff --git a/content/courses/parallel-computing-introduction/distributed_mpi_recvstatus.md b/content/courses/parallel-computing-introduction/distributed_mpi_recvstatus.md new file mode 100644 index 00000000..55f53554 --- /dev/null +++ b/content/courses/parallel-computing-introduction/distributed_mpi_recvstatus.md @@ -0,0 +1,74 @@ +--- +title: "Receive Status" +toc: true +type: docs +weight: 60 +menu: + parallel_programming: + parent: Distributed-Memory Programming +--- + +The _status_ variable contains information about the message. + +* MPI_SOURCE + The source rank of the message. This is a field of the status structure. + * C++ + status.MPI_SOURCE + * Fortran + status(MPI_SOURCE) + * Python + status.Get_Source() + +* MPI_TAG + The tag. This is another field of the structure (or element of the array). + * C++ + status.MPI_TAG + * Fortran + status(MPI_TAG) + * Python + status.Get_tag() + +* MPI_Get_count(MPI_Status\* status, MPI_Datatype datatype, int\* count) + The length (item count) of the message. + * C++ + MPI_Get_count(&status,MPI_TYPE,&item_count); + * Fortran + MPI_Get_count(status,MPI_TYPE,item_count,ierr) + * Python + status.Get_count(MPI.TYPE) + +* MPI_Error + The error number. Not often needed. + * C++ + status.MPI_ERROR + * Fortran + status(MPI_ERROR) + * Python + status.Get_error() + +The MPI_SOURCE and MPI_TAG items may be especially useful for the special dummy variables defined for source and tag. + +## Special Source and Tag Variables + +MPI defines special variables that can be used in MPI_Recv +```no-highlight +MPI_ANY_SOURCE +MPI_ANY_TAG +``` +Either or both can be used in a receive if the MPI_Recv can accept a message from any source and/or any tag. + +**Example Status Usage** + +Add a status query to the exchange code. Note that Python uses functions to retrieve all the members of the status structure, and must also initialize it with a constructor. + +{{< spoiler text="C++" >}} +{{< code-download file="/courses/parallel-computing-introduction/codes/send_recv_stat.cxx" lang="c++" >}} +{{< /spoiler >}} + +{{< spoiler text="Fortran" >}} +{{< code-download file="/courses/parallel-computing-introduction/codes/send_recv_stat.f90" lang="fortran" >}} +{{< /spoiler >}} + +{{< spoiler text="Python" >}} +{{< code-download file="/courses/parallel-computing-introduction/codes/send_recv_stat.py" lang="python" >}} +{{< /spoiler >}} diff --git a/content/courses/parallel-computing-introduction/distributed_mpi_standardsends.md b/content/courses/parallel-computing-introduction/distributed_mpi_standardsends.md index 0d8b2fdf..e18d10d3 100644 --- a/content/courses/parallel-computing-introduction/distributed_mpi_standardsends.md +++ b/content/courses/parallel-computing-introduction/distributed_mpi_standardsends.md @@ -25,6 +25,8 @@ comm.Send([sendbuf,datatype],dest=rank,tag=0) comm.send(sendbuf,dest=rank,tag=0) ``` +## Receive + The matching receive is called as follows: ```c++ @@ -38,35 +40,10 @@ call MPI_Recv(recvbuf, count, datatype, source, int tag, comm, status, ierr) ```python comm.Recv([data,MPI_Type],source=rank,tag=0,status=status) #or for pickled objects -data=comm.Recv(source=rank,tag=0,status=status) +data=comm.recv(source=rank,tag=0,status=status) ``` The status variable is optional for Python, but not for C++ or Fortran. -### Status - -The _status_ variable contains information about the message. - -* MPI_SOURCE - The source rank of the message -* MPI_TAG - The tag -* The length (item count) of the message. This requires another MPI routine. - MPI_Get_count( - MPI_Status* status, MPI_Datatype datatype, int* count) -* The error number - MPI_Error - -The MPI_SOURCE and MPI_TAG items may be especially useful for the special dummy variables defined for source and tag. - -### Special Source and Tag Variables - -MPI defines special variables that can be used in MPI_Recv -```no-highlight -MPI_ANY_SOURCE -MPI_ANY_TAG -``` -Either or both can be used in a receive if the MPI_Recv can accept a message from any source and/or any tag. - **Example** Our first program will run on two processes and exchange some data between them. @@ -82,19 +59,3 @@ Our first program will run on two processes and exchange some data between them. {{< spoiler text="Python" >}} {{< code-download file="/courses/parallel-computing-introduction/codes/send_recv.py" lang="python" >}} {{< /spoiler >}} - -**Example Status Usage** - -Add a status query to the exchange code. Note that Python uses functions to retrieve all the members of the status structure, and must also initialize it with a constructor. - -{{< spoiler text="C++" >}} -{{< code-download file="/courses/parallel-computing-introduction/codes/send_recv_stat.cxx" lang="c++" >}} -{{< /spoiler >}} - -{{< spoiler text="Fortran" >}} -{{< code-download file="/courses/parallel-computing-introduction/codes/send_recv_stat.f90" lang="fortran" >}} -{{< /spoiler >}} - -{{< spoiler text="Python" >}} -{{< code-download file="/courses/parallel-computing-introduction/codes/send_recv_stat.py" lang="python" >}} -{{< /spoiler >}} diff --git a/content/courses/parallel-computing-introduction/solns/a.out b/content/courses/parallel-computing-introduction/solns/a.out index 1e769548..18a4b8f8 100755 Binary files a/content/courses/parallel-computing-introduction/solns/a.out and b/content/courses/parallel-computing-introduction/solns/a.out differ diff --git a/content/courses/parallel-computing-introduction/solns/manager_worker.cxx b/content/courses/parallel-computing-introduction/solns/manager_worker.cxx new file mode 100644 index 00000000..3151dd2d --- /dev/null +++ b/content/courses/parallel-computing-introduction/solns/manager_worker.cxx @@ -0,0 +1,77 @@ +#include +#include +#include +#include +#include +#include +#include +#include "mpi.h" + + +using namespace std; + +int random_int(int n,int m) { + //quick and stupid way, using full C++ machinery is better but complex + return n+rand()%m; +} + +double do_work() { + //hardcoded bounds for convenience + int nsteps=random_int(10000,30000); + float result=0.; + for (int i=0; i results(nprocs); + + int done=0; + float result; + int sender; + + if (rank==0) { + for (int i=1; i +#include + +using namespace std; + +int main(int argc, char **argv) +{ + int nProcs; + int rank; + int baton = 42; + MPI_Status status; + double startTime; + double stopTime; + + MPI_Init(&argc, &argv); + + MPI_Comm_size(MPI_COMM_WORLD, &nProcs); + MPI_Comm_rank(MPI_COMM_WORLD, &rank); + + if (nProcs > 1) + { + if (rank == 0) { + startTime = MPI_Wtime(); + MPI_Send(&baton, 1, MPI_INT, 1, 0, MPI_COMM_WORLD); + MPI_Recv(&baton, 1, MPI_INT, MPI_ANY_SOURCE, + MPI_ANY_TAG, MPI_COMM_WORLD, &status); + cout<<"Process "< +#include "mpi.h" + +using namespace std; + +int main(int argc, char *argv[]) { + int rank, nprocs; + MPI_Status status; + + MPI_Init(&argc, &argv); + MPI_Comm_size(MPI_COMM_WORLD, &nprocs); + MPI_Comm_rank(MPI_COMM_WORLD, &rank); + + if ( nprocs < 2 ) { + cout<<"This program runs only on at least two processes\n"; + MPI_Finalize(); + exit(1); + } + + float message=42.*rank; + + if ( rank == 0 ) { + MPI_Recv(&message,1,MPI_FLOAT,1,0,MPI_COMM_WORLD,&status); + } + else if ( rank == nprocs-1) { + MPI_Send(&message,1,MPI_FLOAT,nprocs-2,0,MPI_COMM_WORLD); + } + else { + MPI_Send(&message,1,MPI_FLOAT,rank-1,0,MPI_COMM_WORLD); + MPI_Recv(&message,1,MPI_FLOAT,rank+1,0,MPI_COMM_WORLD,&status); + } + + if (rank==0) { + cout<<"First Case\n"; + } + + for (int n=0; n +#include "mpi.h" + +using namespace std; + +int main(int argc, char *argv[]) { + int rank, nprocs; + MPI_Status status; + + MPI_Init(&argc, &argv); + MPI_Comm_size(MPI_COMM_WORLD, &nprocs); + MPI_Comm_rank(MPI_COMM_WORLD, &rank); + + if ( nprocs < 2 ) { + cout<<"This program runs only on at least two processes\n"; + MPI_Finalize(); + exit(1); + } + + float message=42.*rank; + + if ( rank == 0 ) { + MPI_Send(&message,1,MPI_FLOAT,nprocs-1,0,MPI_COMM_WORLD); + MPI_Recv(&message,1,MPI_FLOAT,1,0,MPI_COMM_WORLD,&status); + } + else if ( rank == nprocs-1) { + MPI_Send(&message,1,MPI_FLOAT,nprocs-2,0,MPI_COMM_WORLD); + MPI_Recv(&message,1,MPI_FLOAT,0,0,MPI_COMM_WORLD,&status); + } + else { + MPI_Send(&message,1,MPI_FLOAT,rank-1,0,MPI_COMM_WORLD); + MPI_Recv(&message,1,MPI_FLOAT,rank+1,0,MPI_COMM_WORLD,&status); + } + + if (rank==0) { + cout<<"Second Case\n"; + } + + for (int n=0; n +``` + diff --git a/content/notes/rivanna-command-line/creating_editing_files_graphically.md b/content/notes/rivanna-command-line/creating_editing_files_graphically.md new file mode 100644 index 00000000..6de639ce --- /dev/null +++ b/content/notes/rivanna-command-line/creating_editing_files_graphically.md @@ -0,0 +1,30 @@ +--- +title: Creating and Editing Files +date: 2023-12-11-14:11:14Z +type: docs +weight: 900 +menu: + rivanna-command-line: +--- + +To create files we use a _text editor_. Do not use a word processor such as LibreOffice. + +## Graphical Options + +Graphical editors must be used from within a graphical environment. On Linux the standard graphical windowing system is called **X11**. Newer Linux versions provide some form of "desktop" environment similar to Windows or MacOS on top of X11. On Rivanna we provide the [MATE](https://mate-desktop.org/) Desktop Environment. It can be accessed from [FastX](https://www.rc.virginia.edu/userinfo/rivanna/logintools/fastx/) on a frontend. It can also be started on a compute node from the [Open OnDemand](https://www.rc.virginia.edu/userinfo/rivanna/ood/overview/) Desktop Interactive Application. + +### gedit/pluma + +Modern Linux systems provide at least one graphical text editor. One option is `gedit`. On the MATE Desktop used on Rivanna, the equivalent is `pluma`, but the `gedit` command starts pluma. + +Gedit/pluma is very similar to Notepad on Windows. + +### VS Code + +This is accessed by a [module](/notes/rivanna-command-line/modules). +```bash +$module load code-server +``` + +Then open a browser (Firefox is the default on MATE) and go to the URL `http://127.0.0.1:8080`. + diff --git a/content/notes/rivanna-command-line/directory_commands_summary.md b/content/notes/rivanna-command-line/directory_commands_summary.md new file mode 100644 index 00000000..ea37695c --- /dev/null +++ b/content/notes/rivanna-command-line/directory_commands_summary.md @@ -0,0 +1,51 @@ +--- +title: Directory Commands +date: 2023-12-11-14:11:14Z +type: docs +weight: 820 +menu: + rivanna-command-line: +--- + +## cd + +`cd` change directory +```bash +cd +``` +With no argument, `cd` moves to the user's home directory. If you are hopelessly lost, you can always type `cd` and start over. + +```bash +cd ../projects/project1 +``` + +## mkdir + +`mkdir` to create a directory. Works relative to current working directory. +```bash +mkdir new_project +``` +or +```bash +mkdir new_project +mkdir ~/projects/new_project/src +``` + +## rmdir + +`rmdir` to remove an _empty_ directory. Does not affect directories with content. +```bash +rmdir ~/projects/new_project/src +``` + +## rm -rf + +`rm -rf` to force the removal of a directory and all its contents, including subdirectories. + +```bash +rm -rf ~/projects/project1/src +``` + +## cp -r + +Directories are copied with the `cp -r` command previously discussed. diff --git a/content/notes/rivanna-command-line/dotfiles.md b/content/notes/rivanna-command-line/dotfiles.md new file mode 100644 index 00000000..0bc34e74 --- /dev/null +++ b/content/notes/rivanna-command-line/dotfiles.md @@ -0,0 +1,25 @@ +--- +title: Dotfiles +date: 2023-12-11-14:11:14Z +type: docs +weight: 1590 +menu: + rivanna-command-line: +--- + +“Dotfiles” are files that describe resources to programs that look for them. They begin with a period or “dot” (hence the name). In general, they are used for configuration of various software packages. + +Dotfiles are hidden from `ls`, but `ls -a` shows them. Sometimes `ls` is _aliased_ to ls –a. + +Bash has configuration dotfiles: `.bash_profile` and `.bashrc`. + * if no .bash_profile is present it will read .profile + * .bash_profile is sourced only for login shell + * the default .bash_profile on our HPC system incorporates the .bashrc; otherwise on a general Linux system, .bashrc is not run for a login shell. + +Dot "files" may also be, and often are, directories. + +```bash +$ls -a +. .bash_logout .bashrc .lesshst shakespeare .Xauthority +.. .bash_profile data .mozilla .ssh +``` diff --git a/content/notes/rivanna-command-line/exercise_1.md b/content/notes/rivanna-command-line/exercise_1.md new file mode 100644 index 00000000..74c3df0d --- /dev/null +++ b/content/notes/rivanna-command-line/exercise_1.md @@ -0,0 +1,30 @@ +--- +title: Exercise 1 +date: 2023-12-11-14:11:14Z +type: docs +weight: 850 +menu: + rivanna-command-line: +--- + +Start a terminal on a Rivanna frontend by whatever means you prefer. Type (the `$` is the prompt and is not typed): + +```bash +$echo $SHELL +$pwd +$mkdir test_dir +$cd test_dir +$ls .. +$mkdir sub_dir +$ls +``` + +What is the full path of your home directory? + +Delete the `sub_dir` folder. + +Copy the directory +```no-highlight +/share/resources/tutorials/rivanna-cli/shakespeare +``` +into your home directory. diff --git a/content/notes/rivanna-command-line/exercise_2.md b/content/notes/rivanna-command-line/exercise_2.md new file mode 100644 index 00000000..a6624fb5 --- /dev/null +++ b/content/notes/rivanna-command-line/exercise_2.md @@ -0,0 +1,23 @@ +--- +title: Exercise 2 +date: 2023-12-11-14:11:14Z +type: docs +weight: 1100 +menu: + rivanna-command-line: +--- + +Create a directory called `newdir`. Navigate to it. Make a new file +```bash +$cat > mynewfile +``` + +Save it as a new empty file with `^d`. Use nano or another editor of your preference and type a line or two of text. + +```bash +$more mynewfile +$ls +$mv mynewfile the_file +$cp the_file old_file +$ls -l +``` diff --git a/content/notes/rivanna-command-line/file_commands_summary.md b/content/notes/rivanna-command-line/file_commands_summary.md new file mode 100644 index 00000000..b624467a --- /dev/null +++ b/content/notes/rivanna-command-line/file_commands_summary.md @@ -0,0 +1,84 @@ +--- +title: File Commands +date: 2023-12-11-14:11:14Z +type: docs +weight: 800 +menu: + rivanna-command-line: +--- + +## ls + +`ls` list files in a directory. +```bash +ls +``` +With no argument, listing the entire contents of the current working directory is assumed. + +Like most Unix commands, it has many options. They may be combined. + +{{< table >}} +| Option | Purpose | +|-------|-----| +|-l | long listing, includes file date, size, and permissions | +|-a | displays all files including hidden (dotfiles) | +|-h | show file sizes in human readable terms | +|-C | lay out listing in columns | +|-1 | (digit one) list one file per line, no header | +|-t | show the newest files first | +|-r | reverse the order | +|-F |append a symbol to indicate the type of file (ordinary, executable, directory | +{{< /table >}} + +```bash +ls -ltr ./projects +``` + +## cp + +`cp` to copy a file. +```bash +cp file1.txt file2.txt +cp mymod.py ../projects/python_code +``` + +Commonly-used options: + +{{< table >}} +| Option | Purpose | +|-------|-----| +| -i | ask for confirmation before overwriting an existing file | +| -r | copy recursively all subdirectories| +| -n | "noclobber"; do not overwrite an existing file| +| -f | force an overwrite of an existing file| +{{< /table >}} + +```bash +cp -r /share/resources/tutorials/rivanna-cl ~ +``` + +## mv + +`mv` to rename or _move_ a file. +```bash +mv file1.txt file2.txt +mv mymod.py ../projects/python_code +``` + +Options for `mv` are similar to `cp`. + +## rm + +`rm` to remove a file or directory + +```bash +rm file1.txt +rm file1.txt data.csv +``` +Once the file is removed, it is gone and can no longer be accessed. + +Options for `rm` are similar to `cp`. + +{{< warning >}} +By default, the rm command _does not_ ask for confirmation before deleting a file! Use the `-i` option if you are unsure. +{{< /warning >}} diff --git a/content/notes/rivanna-command-line/file_permissions.md b/content/notes/rivanna-command-line/file_permissions.md new file mode 100644 index 00000000..742442cb --- /dev/null +++ b/content/notes/rivanna-command-line/file_permissions.md @@ -0,0 +1,87 @@ +--- +title: File Permissions +date: 2023-12-11-14:11:14Z +type: docs +weight: 860 +menu: + rivanna-command-line: +--- + +Each file and directory has a set of _permissions_ associated with it. Traditionally, Unix has assigned permissions to the _owner_ (the user), the _group_ (a set of users), _others_ (users not the owner nor a member of the group that may have permissions on the file), and _all_ (all users on the system). These are generally denoted by `ugoa`. + +Within each category, a user can have permission to _read_, _write_, or _execute_ the file. These are denoted by `rwx`. + +The command +```bash +ls -l +``` +shows the permissions associated with each file. Since in Unix a directory is a special type of file, the permissions are the same. + +```bash +$ls -l +total 8 +drwxr-xr-x. 2 mst3k mst3k 4096 Dec 18 10:39 data +drwxr-xr-x. 2 mst3k mst3k 4096 Dec 18 10:22 shakespeare + +``` +The output of `ls -l` is + +{{< table >}} +| Permissions | Number of "Hard Links" | Owner | Group | Size | Date and Time Last Modified | Name| +| ---- | ---- | -------- | ------ | ---- | ----------- | ---- | +| drwxr-xr-x | 2 | mst3k | mst3k | 4096 | Dec 18 10:39 | data | +{{< /table >}} + +Permissions not granted to a user are indicated with a hyphen `-`. + +The permissions layout in the first columns should be read as + +{{< table >}} +| Type | Owner | Group Owner | Others | +| ---- | -------- | ------ | ---- | +| d | rwx | r-x | r-x | +{{< /table >}} + +In the above example, the `d` indicates the listing is a directory. A directory must have "execute" `x` permission in order for a user to enter it. + +Listing the files in a directory will result in output such as +```bash +$ls -l shakespeare +-rwxr-xr-x. 1 mst3k mst3k 91 Dec 18 10:22 2col.txt +-rwxr-xr-x. 1 mst3k mst3k 173940 Dec 18 10:22 Hamlet.txt +-rwxr-xr-x. 1 mst3k mst3k 162563 Dec 18 10:22 HamletWords.txt +-rwxr-xr-x. 1 mst3k mst3k 34 Dec 18 10:22 k2sort.txt +-rwxr-xr-x. 1 mst3k mst3k 180857 Dec 18 10:22 Lear.txt +-rwxr-xr-x. 1 mst3k mst3k 154520 Dec 18 10:22 Othello.txt +-rwxr-xr-x. 1 mst3k mst3k 25628 Dec 18 10:22 uniques +-rwxr-xr-x. 1 mst3k mst3k 25628 Dec 18 10:22 uniques.txt +-rwxr-xr-x. 1 mst3k mst3k 72 Dec 18 10:22 wcdemo.txt +-rwxr-xr-x. 1 mst3k mst3k 26 Dec 18 10:22 words_and_num.txt +``` +The hyphen as Type indicates an ordinary file. + +## Changing Permissions + +The owner of a file or directory may change the permissions with the `chmod` command. Two formats are supported: a three- or four-digit code (in octal, i.e. base 8) indicating permissions, or the add/remove symbolic format. The digit format is advanced so we will not discuss it; a reference is available [here](https://www.adminschoice.com/chmod-quick-referance-with-examples). The symbolic format is more intuitive and mnemonic. + +**Examples** + +Add execute permission to a file for its owner. This is frequently used with shell scripts. +```bash +$chmod u+x ascript.sh +``` + +Add execute permissions for the owner and group members +```bash +$chmod ug+x ascript.sh +``` + +Allow others to read and write a file +```bash +$chmod o+wr myfile.txt +``` + +Please note that on multiuser, central systems such as an HPC cluster, the administrators may not allow individual users to change the permissions on certain file sets such as `home` and `scratch` directories, for reasons of data security and privacy. + + + diff --git a/content/notes/rivanna-command-line/file_transfer_from_cl.md b/content/notes/rivanna-command-line/file_transfer_from_cl.md new file mode 100644 index 00000000..472e3a5e --- /dev/null +++ b/content/notes/rivanna-command-line/file_transfer_from_cl.md @@ -0,0 +1,68 @@ +--- +title: Transferring Files from the Command Line +date: 2023-12-11-14:11:14Z +type: docs +weight: 950 +menu: + rivanna-command-line: +--- + +Files can be transferred by [graphical clients](https://www.rc.virginia.edu/userinfo/rivanna/logintools/graphical-sftp) such as MobaXterm and Filezilla, or through [Globus](/tutorials/globus-data-transfer). If you are using a terminal from your local computer, you can also use some command-line tools. + +### scp and sftp + +The secure shell protocol includes two file-transfer command-line tools; `scp` and `sftp`. Sftp is scp with a slightly different interface. + +```bash +scp LOCAL_FILE mst3k@rivanna.hpc.virginia.edu:REMOTE_PLACE +scp mst3k@rivanna.hpc.virginia.edu:REMOTE_FILE LOCAL_PLACE +``` +`REMOTE_PLACE` and `LOCAL_PLACE` refer to the location on the appropriate host where you want the file to be written. `REMOTE_PLACE` can be omitted and the file will be transferred to your home directory under the same name. To change the name or specify a directory on the other system, use a different name or path for `REMOTE_PLACE`. + +`LOCAL_PLACE` must be present but can be `.` for the directory where the `scp` was run. + +The colon `:` is required. + +To copy a directory, use `scp -r` similarly to `cp -r`. + +**Examples** +```bash +scp myfile.txt mst3k@rivanna.hpc.virginia.edu: +scp myscript.py mst3k@rivanna.hpc.virginia.edu:project1 +scp myscript.py mst3k@rivanna.hpc.virginia.edu:script.py +scp myfile.csv mst3k@rivanna.hpc.virginia.edu:/scratch/mst3k +scp mst3k@rivanna.hpc.virginia.edu:/scratch/mst3k/run11/output.txt . +``` + +Scp resembles cp. Sftp is an implementation over scp of the interface of a popular, but insecure, protocol widely used in the past called `ftp` (file transfer protocol). + +```bash +$sftp mst3k@rivanna.hpc.virginia.edu +sftp> put local_file +sftp> get remote_file +sftp> quit +``` +The sftp client permits other commands. `ls` lists files on the remote system. `lls` lists local files. + +### rsync + +The rsync command is used to _synchronize_ files and folders. It has many options and some attention must be paid to whether a trailing slash is needed or not. + +```bash +rsync -av ldir/ mst3k@rivanna.hpc.virginia.edu:rdir +rsync my_file mst3k@rivanna.hpc.virginia.edu:/scratch/$USER +``` +By default `rsync` does not transfer files that are older than the equivalent on the target. This can increase the transfer speed significantly. Rsync also resumes a transfer that was interrupted. Scp always starts again from the beginning. + +Rsync is very powerful but has many options and can be confusing. For more details see our [documentation](https://www.rc.virginia.edu/userinfo/rivanna/logintools/cl-data-transfer). Several online resources with examples are also available, such as [this](https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories). + +### Setting up Passwordless ssh + +If you will frequently use ssh, scp, sftp, or rsync to a remote system, it becomes inconvenient to repeatedly enter your password. You can generate a _key_ to log in without a password. + +1. Go to the `.ssh` directory. Note that this is a hidden directory. +2. Run the command `ssh-keygen` +3. When asked for a passcode, hit `Enter` to leave it blank. +4. Transfer the file `id_rsa.pub` to the remote system. You may wish to name it something else, such as `myhost.pub`. +5. Append this file ending in .pub to a file called `authorized_keys`. Create it if it doesn't exist. + diff --git a/content/notes/rivanna-command-line/files_and_directories.md b/content/notes/rivanna-command-line/files_and_directories.md new file mode 100644 index 00000000..628d7a4b --- /dev/null +++ b/content/notes/rivanna-command-line/files_and_directories.md @@ -0,0 +1,31 @@ +--- +title: Files and Directories +date: 2023-12-11-14:11:14Z +type: docs +weight: 700 +menu: + rivanna-command-line: +--- + +**Files** store some sort of information: data, source code for scripts or programs, images, video, and so forth. + +There are two basic types of files: + * Text (documents, code) + * Binary (images, executables) + +The Unix shells ignore file extensions, but software might require certain extensions. This includes compilers (`.cxx`, `.f90` and so forth), interpreters (`.py`, `.R`, etc.), image and video tools (`.png`, `.jpg`, `.mp4`). Since the format of the file extension depends on the expectations of software or on user preference, there is no rule limiting the number of characters, but most consist of one to three characters. + +**Directories** are collections of files and other directories. Often they are called _folders_, but directory is generally the preferred (and historical) term in Unix. + +Directories that are stored within another directory are _subdirectories_ of the parent. + +Both files and directories have “metadata” associated with them such as name, timestamps, and permissions. + +The names of Unix files and directories should contain _no spaces_. +```bash +$ls + +'10. directories.md' +'15. streams.md' +``` +The quotes indicate that a space is part of the file name. While most modern Unix tools can handle spaces, the shell does not always do so, and special precautions must be taken to avoid suprises. For that reason, underscores or hyphens are preferred instead of spaces. diff --git a/content/notes/rivanna-command-line/getting_command_information.md b/content/notes/rivanna-command-line/getting_command_information.md new file mode 100644 index 00000000..3fc6bdc6 --- /dev/null +++ b/content/notes/rivanna-command-line/getting_command_information.md @@ -0,0 +1,22 @@ +--- +title: Finding Information About a Command +date: 2023-12-11-14:11:14Z +type: docs +weight: 1050 +menu: + rivanna-command-line: +--- + +Basic documentation for a command can be read from the shell with the `man` (manual) command. +```bash +$man ls +``` + +Many commands also provide a `--help` option which usually prints the same information. +```bash +$ls --help +``` + +The output of man is often called the _manpage_ of that command. + + diff --git a/content/notes/rivanna-command-line/getting_help.md b/content/notes/rivanna-command-line/getting_help.md new file mode 100644 index 00000000..297309f3 --- /dev/null +++ b/content/notes/rivanna-command-line/getting_help.md @@ -0,0 +1,10 @@ +--- +title: Need help? +date: 2023-12-11-14:11:14Z +type: docs +weight: 3000 +menu: + rivanna-command-line: +--- + +Research Computing is ready to help you learn to use our systems efficiently. You can [submit a ticket](https://www.rc.virginia.edu/form/support-request/). For in-person help, please attend one of our weekly sessions of [office hours](https://www.rc.virginia.edu/support/#office-hours). diff --git a/content/notes/rivanna-command-line/grep.md b/content/notes/rivanna-command-line/grep.md new file mode 100644 index 00000000..66a6e36e --- /dev/null +++ b/content/notes/rivanna-command-line/grep.md @@ -0,0 +1,60 @@ +--- +title: Finding and Searching Files +date: 2023-12-11-14:11:14Z +type: docs +weight: 1360 +menu: + rivanna-command-line: +--- + +## Searching with grep + +The `grep` command is commonly used in UNIX to filter a file or input, line by line, against a pattern. Patterns can be complex and use _regular expressions_, but most of the time wildcards are sufficient. + +```no-highlight +grep [OPTIONS] PATTERN FILENAME +``` +**Example** + +The `-i` option stands for "ignore case." +```bash +$grep -i Unix intro_basic-unix.txt +``` + +Grep is frequently used with wildcards and pipes. +```bash +$grep -i write *f90 +$grep weight: *md | grep 100 +``` + +**Example** + +How many sequences are in a FASTA-formatted file? Each sequence record in a FASTA file has one line of description that always starts with `>`, followed by multiple lines of the sequence itself. Each sequence record ends when the next line starts with `>`. + +```bash +$grep -c '>' sequences.fasta +``` +The `-c` option returns the number of lines containing the pattern. Please be sure to include the quotes around the `>` or the shell will interpret it as a redirection. + +**A Handy Trick** + +To find all occurrences of a pattern in all files in a directory, use `grep -r`. +```bash +$grep -r "print" python_programs +``` +Be careful with the pattern for a recursive search, or the output can be excessive. + +## Finding Files by Name + +The `find` command can locate a file if you cannot remember its directory. It can take wildcards, in which case it is best to use quotes around the name pattern. + +```bash +$find . -name 2col.txt +./shakespeare/2col.txt +$find . -name "people*" +./data/people.txt +``` +The period `.` tells find to start at the current working directory. + +Find has many options to locate files by name, type, date, and others. See [here](https://www.tecmint.com/35-practical-examples-of-linux-find-command/) for examples. + diff --git a/content/notes/rivanna-command-line/hpc_environment.md b/content/notes/rivanna-command-line/hpc_environment.md new file mode 100644 index 00000000..bc12237a --- /dev/null +++ b/content/notes/rivanna-command-line/hpc_environment.md @@ -0,0 +1,12 @@ +--- +title: HPC Specifics +date: 2023-12-11-14:11:14Z +type: docs +weight: 1600 +menu: + rivanna-command-line: +--- + +Most high-performance computing clusters have commands that are specific to that environment, and are not part of Unix per se. + + diff --git a/content/notes/rivanna-command-line/index.md b/content/notes/rivanna-command-line/index.md deleted file mode 100644 index 28ee3940..00000000 --- a/content/notes/rivanna-command-line/index.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: "Using Rivanna from the Command Line" -weight: 2000 ---- -This workshop provides an introduction to the command-line interface. We’ll learn how to use commands to perform basic operations in the terminal — creating or navigating directories, listing and displaying files, moving or copying files — as well as searching files, managing file permissions, and creating symbolic links. Working in the command line, we can combine existing programs, automate repetitive tasks, and connect to remote resources. - -{{< slideshow folder="slides/rivanna-command-line" ext="png" >}} diff --git a/content/notes/rivanna-command-line/logging_in.md b/content/notes/rivanna-command-line/logging_in.md new file mode 100644 index 00000000..b42bee49 --- /dev/null +++ b/content/notes/rivanna-command-line/logging_in.md @@ -0,0 +1,26 @@ +--- +title: Logging In +date: 2023-12-11-14:11:14Z +type: docs +weight: 300 +menu: + rivanna-command-line: +--- + +Logging into a remote UNIX based system requires a program generally called a _client_. The options for the client depends on your OS. + +## SSH + +Command line access through a terminal on your computer is based on the `ssh` or Secure Shell protocol. It is + * Encrypted + * Available on most UNIX systems + +Your ssh client communicates with the ssh _server_ program running on the remote system. Once established, the connection is secure even if the network is insecure. + +**Example** + +If your computer is a MacOS or Linux system, you log in with +```bash +ssh -Y mst3k@virginia.edu +``` +Throughout this tutorial we will use `mst3k` as our example user ID. You should substitute your own. The option `-Y` allows access to graphical applications and requires that an _X11 server_ application must be installed on your computer. This should be the default for Linux, but MacOS users must install [XQuartz](https://xquartz.org) before this command-line option will be useful. diff --git a/content/notes/rivanna-command-line/logging_in_graphically.md b/content/notes/rivanna-command-line/logging_in_graphically.md new file mode 100644 index 00000000..82b32e1b --- /dev/null +++ b/content/notes/rivanna-command-line/logging_in_graphically.md @@ -0,0 +1,23 @@ +--- +title: Logging in With Graphical Applications +date: 2023-12-11-14:11:14Z +type: docs +weight: 350 +menu: + rivanna-command-line: +--- + +The command-line secure shell is not the only option for accessing Rivanna. Windows users in particular may wish to use other methods, since although ssh is available for it, Windows is not particularly friendly to command lines. + +**Open OnDemand** (OOD) + +Open OnDemand is a Web-based interface to the system. It provides a graphical file-management interface and access to several popular applications running on the compute nodes. A simple terminal that opens on a frontend is also provided. +See the introduction in our basic [tutorial](/notes/rivanna-intro/connecting_to_rivanna/connecting_ood). + +**FastX** + +FastX is a Web-based graphical interface to a frontend. It is also covered in the [introduction](/notes/rivanna-intro/connecting_to_rivanna/connecting_fastx). + +**MobaXterm** (Windows) + +MobaXterm combines an ssh client, an _sftp_ client for file transfers, and an X11 server into a single bundle. More details are available at our [Website](https://www.rc.virginia.edu/userinfo/rivanna/logintools/mobaxterm/) or in the [introduction](/notes/rivanna-intro/files/file_moba). diff --git a/content/notes/rivanna-command-line/managing_processes.md b/content/notes/rivanna-command-line/managing_processes.md new file mode 100644 index 00000000..a04c4a23 --- /dev/null +++ b/content/notes/rivanna-command-line/managing_processes.md @@ -0,0 +1,78 @@ +--- +title: Process Control +date: 2023-12-11-14:11:14Z +type: docs +weight: 1500 +menu: + rivanna-command-line: +--- + +A running executable is a _process_ to the Unix operating system. When it is run at a command line, a process can be running in the _foreground_, which suppresses a prompt, or in the _background_, which returns a prompt to the shell. + +To start in the background, add an ampersand `&` at the end of the command. +```bash +$./myexec -o myopt myfile& +``` + +## Managing Processes + +The `jobs` command lists your running jobs (processes) with their job index. + +The key combination control-z (ctrl-z or ^z) suspends the foreground job. To resume the job in the background, type `bg`. + +This can be combined with output from `jobs` +```bash +$bg %1 # place the job number 1 into the background +$fg %4 # place the job number 4 back to the foreground +``` + +For more general information about processes, use `ps` (process status) The `-u` option limits it to processes owned by user `mst3k`. +```bash +$ps -u mst3k + PID TTY TIME CMD +498571 ? 00:00:00 systemd +498575 ? 00:00:00 (sd-pam) +498581 ? 00:00:00 pulseaudio +498582 ? 00:00:00 sshd +498593 pts/3 00:00:00 bash +498665 ? 00:00:00 dbus-daemon +498670 ? 00:00:00 dbus-daemon +498672 ? 00:00:00 dbus-kill-proce +498677 ? 00:00:00 gio +498685 ? 00:00:00 gvfsd +498691 ? 00:00:00 gvfsd-fuse +517189 pts/3 00:00:00 ps +``` + +The `pid` is the _process id_. + + +## Killing Processes + +You have accidentally started a production job on a frontend node. What to do? + +You can kill your forground process with `Crtl c`. +```bash +#oops, I was supposed to run this through Slurm +$./myexe +^c +``` + +If you need to kill a background process, you can use `jobs` to locate and foreground it. You may also have processes that don't appear with `jobs`. Use `ps` to find the PID, then +```bash +$kill -9 +``` +Do not type the angle brackets, just the number. Many processes will ignore the kill command without the `-9` option so we routinely include it. + +To kill by executable name +```bash +$killall -9 +``` +The kill command with -9 immediately kills the process without allowing the process to clean up or save data. The killall command can be used to kill all of the processes that match a specific name or pattern. + +If you find yourself in a jam and do not know what is wrong and you must start over, +```bash +$kill -9 -1 +``` +kills **all** your processes, including your login. + diff --git a/content/notes/rivanna-command-line/modules.md b/content/notes/rivanna-command-line/modules.md new file mode 100644 index 00000000..09b2ecf5 --- /dev/null +++ b/content/notes/rivanna-command-line/modules.md @@ -0,0 +1,15 @@ +--- +title: The Modules Environment +date: 2023-12-11-14:11:14Z +type: docs +weight: 1900 +menu: + rivanna-command-line: +--- + +_Environment modules_ are not strictly a part of Unix, but are widely used by many HPC sites, including ours. Modules enable the user to set complex paths, environment variables, and so forth, simply by loading a module. + +The commands are set up automatically when you log in. Loaded modules only affect the shell in which the command is run. + +Modules required for a job must be loaded in the batch _job script_. + diff --git a/content/notes/rivanna-command-line/modules_commands.md b/content/notes/rivanna-command-line/modules_commands.md new file mode 100644 index 00000000..d152bc9c --- /dev/null +++ b/content/notes/rivanna-command-line/modules_commands.md @@ -0,0 +1,29 @@ +--- +title: Modules Commands +date: 2023-12-11-14:11:14Z +type: docs +weight: 2600 +menu: + rivanna-command-line: +--- + +{{< table >}} +| Command | Result | +| ------- | ------ | +| module load | load the default module for the package | +| module load / | load the specific version of a package | +| module spider | view the available versions of a package | +| module spider / | view instructions for loading a version | +| module purge | clear all modules | +| module swap / / | switch from version 1 to version 2 | +{{< /table >}} + +**Examples** +```bash +$module load gcc +$module load matlab/R2023a +$module spider R/4.3.1 +$module load gcc/11.4.0 openmpi/4.1.4 R/4.3.1 +$module purge +``` + diff --git a/content/notes/rivanna-command-line/navigating_the_shell.md b/content/notes/rivanna-command-line/navigating_the_shell.md new file mode 100644 index 00000000..e3514e10 --- /dev/null +++ b/content/notes/rivanna-command-line/navigating_the_shell.md @@ -0,0 +1,43 @@ +--- +title: Navigating the Bash Shell +date: 2023-12-11-14:11:14Z +type: docs +weight: 550 +menu: + rivanna-command-line: +--- + +Modern shells provide useful "hotkey" commands that can save considerable typing. The control- notation means that the `Ctrl` (control) key and the following key should be depressed at the same time. + +{{< table >}} + +| Function | Key | +|-------|-----| +|Tab completion: expand typed string to as far as it has a unique name. | tab | +|Search for earlier command | control-r | +|Move to the beginning of the line | control-a | +|Move to the end of the line | control-e | +|Clear the screen | `clear` or control-l | +{{< /table >}} + +## Arrow Keys + +When using bash you may use its built-in history mechanism to save yourself some keystrokes. + +{{< table >}} +| Function | Key | +|-------|-----| +|scroll through the previous commands you have typed | up arrow| +|if already scrolled back, scroll to more recent commands | down arrow | +|edit text on a line | right or left arrow | +{{< /table >}} + +## Logging Out + +To log out of a shell, type `exit` +```bash +exit +``` + +This logs you out of your current terminal window. If that is a login window, for example your `ssh` shell, it will log you out of the system entirely. + diff --git a/content/notes/rivanna-command-line/newfile b/content/notes/rivanna-command-line/newfile new file mode 100644 index 00000000..cfad7120 --- /dev/null +++ b/content/notes/rivanna-command-line/newfile @@ -0,0 +1,3 @@ +Now I can enter some text +it will be copied exactly +into the file diff --git a/content/notes/rivanna-command-line/other_file_viewing_tools.md b/content/notes/rivanna-command-line/other_file_viewing_tools.md new file mode 100644 index 00000000..7f28192a --- /dev/null +++ b/content/notes/rivanna-command-line/other_file_viewing_tools.md @@ -0,0 +1,98 @@ +--- +title: Other File Viewing Tools +date: 2023-12-11-14:11:14Z +type: docs +weight: 935 +menu: + rivanna-command-line: +--- + +## Printing a file to the console + +The `cat` (concatenate) command prints the contents of a file to the screen. +```bash +cat myfile +``` +It does not fit the output to the terminal windows size; it simply keeps printing to the end. + +It can also be used to create or join files (hence its name). We will learn more about its behavior here when we look at [standard streams](/notes/rivanna-command-line/standard_streams). + +```bash +#create a file +cat > newfile +Now I can enter +some text +It will be copied exactly into the file +^d +``` +The `^d` notation means hold down `Ctrl` and `d` together. It is the end-of-file marker for bash. + +```bash +#append a file to another +cat otherfile >> newfile +``` +{{< warning >}} +If the double `>>` is omitted, `cat` will overwrite `newfile` with `otherfile`. +{{< /warning >}} + +## Displaying Parts of a File + +### head and tail + +`head` filename + +Displays only the starting lines of a file. The default is first ten lines. Use “-n” to specify the number of lines. +```bash +$head ~/rivanna-cli/shakespeare/Lear.text +``` +
+This Etext file is presented by Project Gutenberg, in
+cooperation with World Library, Inc., from their Library of the
+Future and Shakespeare CDROMS.  Project Gutenberg often releases
+Etexts that are NOT placed in the Public Domain!!
+
+*This Etext has certain copyright implications you should read!*
+
+<<THIS ELECTRONIC VERSION OF THE COMPLETE WORKS OF WILLIAM
+SHAKESPEARE IS COPYRIGHT 1990-1993 BY WORLD LIBRARY, INC., AND IS
+PROVIDED BY PROJECT GUTENBERG WITH PERMISSION.  ELECTRONIC AND
+
+ +`tail` filename + +Displays the last 10 lines. +```bash +$tail 30 ~/rivanna-cli/shakespeare/Lear.text +``` +
+     The cup of their deservings.- O, see, see!
+  Lear. And my poor fool is hang'd! No, no, no life!
+     Why should a dog, a horse, a rat, have life,
+     And thou no breath at all? Thou'lt come no more,
+     Never, never, never, never, never!
+     Pray you undo this button. Thank you, sir.
+     Do you see this? Look on her! look! her lips!
+     Look there, look there!                            He dies.
+  Edg. He faints! My lord, my lord!
+  Kent. Break, heart; I prithee break!
+  Edg. Look up, my lord.
+  Kent. Vex not his ghost. O, let him pass! He hates him
+     That would upon the rack of this tough world
+     Stretch him out longer.
+  Edg. He is gone indeed.
+  Kent. The wonder is, he hath endur'd so long.
+     He but usurp'd his life.
+  Alb. Bear them from hence. Our present business
+     Is general woe. [To Kent and Edgar] Friends of my soul, you
+        twain
+     Rule in this realm, and the gor'd state sustain.
+  Kent. I have a journey, sir, shortly to go.
+     My master calls me; I must not say no.
+  Alb. The weight of this sad time we must obey,
+     Speak what we feel, not what we ought to say.
+     The oldest have borne most; we that are young
+     Shall never see so much, nor live so long.
+                                       Exeunt with a dead march.
+
+THE END
+
diff --git a/content/notes/rivanna-command-line/our_first_command.md b/content/notes/rivanna-command-line/our_first_command.md new file mode 100644 index 00000000..7f187b56 --- /dev/null +++ b/content/notes/rivanna-command-line/our_first_command.md @@ -0,0 +1,15 @@ +--- +title: Our First Command +date: 2023-12-11-14:11:14Z +type: docs +weight: 500 +menu: + rivanna-command-line: +--- + +Let’s run our first command. Into a terminal type +```bash +$pwd +/home/mst3k +``` +This command stands for *p*rint *w*orking *d*irectory. It prints the name of the folder in which the shell is currently working. diff --git a/content/notes/rivanna-command-line/paths.md b/content/notes/rivanna-command-line/paths.md new file mode 100644 index 00000000..d125a3cf --- /dev/null +++ b/content/notes/rivanna-command-line/paths.md @@ -0,0 +1,55 @@ +--- +title: Paths +date: 2023-12-11-14:11:14Z +type: docs +weight: 750 +menu: + rivanna-command-line: +--- + +In most operating systems, all files and directories are located with a **path**. The “path” is the "full name" of every file & directory. + +In a Unix-based operating system, the files are organized into a _tree_ structure. The tree is "upside down" because the **root directory** is at the top, and directories branch off from there. The root directory is indicated with a forward slash `/`. + +{{< diagram >}} +graph TD + C["/"] --> etc["etc"] + C["/"] --> opt["opt"] + C["/"] --> lib["lib"] + C --> usr["usr"] + C --> home["home"] + C --> var["var"] + + home --> mst3k["mst3k"] + home --> im4u["im4u"] + + mst3k-->myfile["myfile"] + mst3k-->mydir["mydir"] + + mydir-->file1["file1"] + mydir-->file2["file2"] + + usr--> bin["bin"] + usr--> share["share"] + usr--> local["local"] + + bin--> ls["ls"] + bin--> pwd["pwd"] + bin--> cd["cd"] +{{< /diagram >}} + +The **home** directory is the usual name on Linux and similar Unix-based operating systems for the folder that holds user directories and files. On MacOS it is called User. On both, the _separator_ between branches of the tree is the forward slash. + +{{< hl >}} +/home/mst3k/myfile +{{< /hl >}} +
+{{< hl >}} +/Users/Misty Tea/Documents/Homework.pages +{{< /hl >}} + +Windows files and folders also have paths. In Windows, drive letters or _volumes_ are the top-level folders, and usually there is more than one. User files are in Users, similar to MacOS. + +{{< hl >}} +C:\Users\Misty Tea\Documents\Homework.docx +{{< /hl >}} diff --git a/content/notes/rivanna-command-line/pipes.md b/content/notes/rivanna-command-line/pipes.md new file mode 100644 index 00000000..f5b70ee8 --- /dev/null +++ b/content/notes/rivanna-command-line/pipes.md @@ -0,0 +1,23 @@ +--- +title: Pipes +date: 2023-12-11-14:11:14Z +type: docs +weight: 1350 +menu: + rivanna-command-line: +--- + +One of the most powerful properties of Unix is that you can __pipe__ the standard output of one command into the standard input of another. + +The pipe symbol `|` is above the backslash on most US keyboards. Pipes can be chained indefinitely, though most uses need just one. +```no-highlight +cmd1 | cmd2 | cmd3 +``` + +**Example** + +Commands such as `ls` have lengthy manpages that will tend to scroll off our terminal window. +```bash +$man ls | more +``` +Now we can page through the listing. diff --git a/content/notes/rivanna-command-line/running_executables.md b/content/notes/rivanna-command-line/running_executables.md new file mode 100644 index 00000000..5efc2ab2 --- /dev/null +++ b/content/notes/rivanna-command-line/running_executables.md @@ -0,0 +1,36 @@ +--- +title: Running Executables +date: 2023-12-11-14:11:14Z +type: docs +weight: 1400 +menu: + rivanna-command-line: +--- + +Executables are often also called _binaries_. The terms are synonymous in most cases. + +The shell has a predefined _search path_. It will look in a sequence of directories for an executable with the specified name, and will invoke the first one it encounters. If the executable is in this search path, you can simply type its name at the prompt. + +```bash +$gedit hello_world.sh +``` + +here `gedit` is the name of the executable. Its actual location is /usr/bin/gedit, but /usr/bin is in the default search path. + +If the location of the binary is not in your search path, you must type the path to the executable (it can be absolute or relative) + +```bash +$./hello_world.sh +``` + +For security reasons the current directory is not in your default search path. Hence you must explicitly provide the `./` path to execute a binary in the current directory. If you wish to add it, type (for bash) +```bash +$export PATH=$PATH:. +``` +`PATH` is called an _environment variable_. It holds the list of paths to be searched by the shell. In this example it is essential to add the first `$PATH` or you will lose the default path set by the system. + +If you are unsure of the path to the binary you wish to run, the `which` command will tell you the path to the binary the shell will start. +```bash +$which g++ +/apps/software/standard/core/gcc/11.4.0/bin/g++ +``` diff --git a/content/notes/rivanna-command-line/running_jobs.md b/content/notes/rivanna-command-line/running_jobs.md new file mode 100644 index 00000000..63b9545a --- /dev/null +++ b/content/notes/rivanna-command-line/running_jobs.md @@ -0,0 +1,14 @@ +--- +title: Running Jobs +date: 2023-12-11-14:11:14Z +type: docs +weight: 2800 +menu: + rivanna-command-line: +--- + +In an HPC environment, the tasks that users want to perform are generically called **jobs**. These must not be confused with the "jobs" of the Unix `jobs` command, although the concepts are related. An HPC "job" is a combination of _resource requests_, any setup required such as loading modules, and the actual commands required to run the desired software. + +HPC jobs are run on _compute nodes_, not on the interactive frontends. + +In the Introduction we discussed using the OOD [job composer](/notes/rivanna-intro/features_of_ood/features_jobs_tab) to submit and monitor jobs. When working at the command line, we must use more detailed features of Slurm. Please go through our [tutorial](/notes/slurm-from-cli) to learn how to write and submit Slurm scripts from the command line. diff --git a/content/notes/rivanna-command-line/running_shell_commands.md b/content/notes/rivanna-command-line/running_shell_commands.md new file mode 100644 index 00000000..ce5581c0 --- /dev/null +++ b/content/notes/rivanna-command-line/running_shell_commands.md @@ -0,0 +1,24 @@ +--- +title: Running Shell Commands +date: 2023-12-11-14:11:14Z +type: docs +weight: 400 +menu: + rivanna-command-line: +--- + +The syntax of Unix commands is not completely standardized but in general follow the pattern of a two or three-letter abbreviation followed by _command-line options_, which are single-letter options preceded by one hyphen or multiple-letter options with two hyphens. Many commands also take _arguments_. + +{{< hl >}} +cmd -o opt1 --anoption opt2 argument +{{}} + +**Example** +Invoke the utility `rm` to delete a file. +```bash +rm myfile +``` +In this example, the shell issues a request to the kernel to delete `myfile`. The kernel then communicates with the software that manages file storage to exectute the operation. + +When it is complete the shell then returns the UNIX prompt to the user, indicating that it is waiting for further commands. + diff --git a/content/notes/rivanna-command-line/standard_streams.md b/content/notes/rivanna-command-line/standard_streams.md new file mode 100644 index 00000000..e4d4e6fa --- /dev/null +++ b/content/notes/rivanna-command-line/standard_streams.md @@ -0,0 +1,22 @@ +--- +title: Standard Streams +date: 2023-12-11-14:11:14Z +type: docs +weight: 1250 +menu: + rivanna-command-line: +--- + +Each executable has associated with it three input/output streams: __standard input__ , __standard error__ , and __standard output__. Normally these streams come from or go to your console (i.e. your terminal). + +Most Unix commands read from standard input and/or write to standard output. + +These I/O streams are often represented as __stdin__, __stderr__, and __stdout__. + +The Unix commands we have studied so far all write to standard output. + +```bash +ls -l +``` +produces output to the terminal. + diff --git a/content/notes/rivanna-command-line/storage_on_hpc.md b/content/notes/rivanna-command-line/storage_on_hpc.md new file mode 100644 index 00000000..7b36b8ac --- /dev/null +++ b/content/notes/rivanna-command-line/storage_on_hpc.md @@ -0,0 +1,18 @@ +--- +title: Your Home Directory +date: 2023-12-11-14:11:14Z +type: docs +weight: 1700 +menu: + rivanna-command-line: +--- + +The default home directory on Rivanna provides 50GB of storage capacity, e.g., /home/mst3k. Each user also has access to 10 TB of __temporary__ storage in the "scratch" folder, e.g. /scratch/mst3k + +The /home and /scratch directories are for personal use and not shareable with other users. + +{{< warning >}} +/scratch is NOT permanent storage and files that have not been accessed for more than 90 days will be marked for deletion. +{{< /warning >}} + +Research groups can lease permanent [storage](https://www.rc.virginia.edu/userinfo/storage/). diff --git a/content/notes/rivanna-command-line/storage_quota.md b/content/notes/rivanna-command-line/storage_quota.md new file mode 100644 index 00000000..12c8eef3 --- /dev/null +++ b/content/notes/rivanna-command-line/storage_quota.md @@ -0,0 +1,22 @@ +--- +title: Checking your Storage +date: 2023-12-11-14:11:14Z +type: docs +weight: 1750 +menu: + rivanna-command-line: +--- + +To see how much disk space, you have used in your home and scratch directories, open a terminal window and type `hdquota` at the command-line prompt: + +```bash +$hdquota +Type    Location         Name             Size Used Avail Use% +======================================================================= +home       /home        mst3k                   51G   12G   39G  24% +Scratch    /project     slurmtests             2.0P  1.9P  144T  93% +Project    /project     arcs                    16T   12T  3.8T  75%   Project    /project     rivanna_software       1.1T  4.2M  1.0T   1%   Project    /project     ds5559                  51G  3.7G   47G   8%   Value      /nv          vol174                 5.5T  1.2T  4.4T  21% +``` + +Note: only home, and other storage just for some + diff --git a/content/notes/rivanna-command-line/stream_redirection.md b/content/notes/rivanna-command-line/stream_redirection.md new file mode 100644 index 00000000..a946a024 --- /dev/null +++ b/content/notes/rivanna-command-line/stream_redirection.md @@ -0,0 +1,34 @@ +--- +title: Standard Stream Redirection +date: 2023-12-11-14:11:14Z +type: docs +weight: 1300 +menu: + rivanna-command-line: +--- + +The standard streams are normally associated with the console. The command will print to the terminal, or will read input typed into the terminal. Stdin and stout can also be _redirected_ to write to or read from a file. + +Redirect standard input with `<` +```bash +$./mycode < params.txt +``` + +Redirect standard output with `>` +```bash +$ls –l > filelist.txt +``` + +If the file exists, `>` will overwrite it. Append with `>>`. +```bash +$cat file1 >> bigfile.csv +``` + +Redirection of standard error depends on the shell and is needed for only a few commands. + +For bash +```bash +$make >& make.out +``` +redirects both stdout and stderr from the `make` command to `make.out`. + diff --git a/content/notes/rivanna-command-line/text_based_editors.md b/content/notes/rivanna-command-line/text_based_editors.md new file mode 100644 index 00000000..8d425dc8 --- /dev/null +++ b/content/notes/rivanna-command-line/text_based_editors.md @@ -0,0 +1,38 @@ +--- +title: Text-Based File Editors +date: 2023-12-11-14:11:14Z +type: docs +weight: 910 +menu: + rivanna-command-line: +--- + +Regular Linux users are advised to learn at least one text-based editor, in case a graphical interface is not available or is too slow. + +### vi (vim) + +The oldest and most widely available text-based editor on Unix is _vi_ for "visual." Modern versions are nearly all based on [vim](https://vim.org) ("vi improved). On Rivanna we generally _alias_ the `vi` command to `vim`. + +Vim/vi is used entirely through keyboard commands which must be memorized. The mouse is not utilized. Only a few commands are needed to be able to do basic editing and they are not difficult to learn. A beginner tutorial is [here](https://www.tutorialspoint.com/vim/index.htm). One stumbling block for new users is that vim has _command mode_ and _insert mode_ and it must be toggled between the two. + +Basics: + * To enter the insert mode press `i` + * To enter the command mode press `ESC` + * To save the file enter `:w` + * To save under a new name `:w filename` + * To exit `:q` + +MATE also provides a graphical interface called `gvim`. It can be accessed through the Applications->Accessories menu as Vi IMproved. GVim combines the keyboard-oriented commands of vim with some mouse-based capabilities. + +### nano + +Nano is simple text editor that is fairly self-explanatory and simple to learn. It is always in insert mode. A tutorial similar to that for vim above is [here](https://www.tutorialspoint.com/how-to-use-nano-text-editor). + +Basics: + * Immediately start typing + * To exit: __control+X__ + * Type the filename and press __Enter__ + +### Emacs + +The Emacs editor is another long-time editor on Unix. Similar to gvim, it combines a graphical interface, when available, with a keyboard-based command system. If the graphical interface is not available, such as from an Open OnDemand terminal, it will fall back to a text-only interface. Emacs is powerful and complex. The documentation is [here](https://www.gnu.org/software/emacs/tour/). diff --git a/content/notes/rivanna-command-line/the_shell.md b/content/notes/rivanna-command-line/the_shell.md new file mode 100644 index 00000000..0817f520 --- /dev/null +++ b/content/notes/rivanna-command-line/the_shell.md @@ -0,0 +1,20 @@ +--- +title: The Shell +date: 2023-12-11-14:11:14Z +type: docs +weight: 250 +menu: + rivanna-command-line: +--- +In all version of Unix, the __shell__ is a program that interprets commands and acts as an interface between the user and the kernel. + +Multiple shells are available. In Linux systems, the default is the `bash` shell. MacOS formerly defaulted to bash as well, but has recently switch to `zsh`. + +The shell displays a __prompt__, which indicates that it is ready to accept commands. In this tutorial, we will utilize the dollar sign `$` as the prompt; yours may be different, and later in this tutorial we will learn how to customize it to your preferences. + +To determine your current shell, at the prompt type +```bash +echo $SHELL +``` +It is important to keep in mind that Unix in general and the shell in particular is __case-sensitive__. The commands `LS` and `ls` would not be the same. + diff --git a/content/notes/rivanna-command-line/the_unix_os.md b/content/notes/rivanna-command-line/the_unix_os.md new file mode 100644 index 00000000..df5901a1 --- /dev/null +++ b/content/notes/rivanna-command-line/the_unix_os.md @@ -0,0 +1,17 @@ +--- +title: The Unix Operating System +date: 2023-12-11-14:11:14Z +type: docs +weight: 200 +menu: + rivanna-command-line: +--- + +__UNIX__ : is a text-oriented operating system (OS) originally developed at Bell Labs during the 1960s. Two versions of this OS are dominant today, _Linux_ and _Mac OS_. + +Strictly speaking, "Linux" refers just to the _kernel_, which is the part of an operating system that manages the hardware interfaces. On top of the kernel sits a number of utilities that enable users and applications to interact with the kernel. + +Linux is the operating system most widely used at HPC facilities, internet servers, and the majority of financial trading system worldwide. A version of Linux powers Android systems. + +Mac OS is based on a slightly different version of Unix. + diff --git a/content/notes/rivanna-command-line/viewing_file_contents.md b/content/notes/rivanna-command-line/viewing_file_contents.md new file mode 100644 index 00000000..91daded5 --- /dev/null +++ b/content/notes/rivanna-command-line/viewing_file_contents.md @@ -0,0 +1,42 @@ +--- +title: Viewing Files +date: 2023-12-11-14:11:14Z +type: docs +weight: 930 +menu: + rivanna-command-line: +--- + +## Pagers + +The most widely used way to view files quickly without editing is to use a _pager_. A pager prints to the terminal window the amount of text that fits in that window. The default pager on Linux is `more` (also called `less` because "less is more"). + +```bash +more filename +``` +This displays file the contents on the screen with line scrolling. To scroll you can use ‘arrow’ keys. To advance one line, press the Enter key. To advance a full page, press the space bar. Press `q` to exit. + +```bash +$more ~/rivanna-cli/shakespeare/Lear.txt +``` + +To page upward within the text, press `b` (back). + +### Searching. + +You can search in the forward direction with `/`, where pattern is a combination of characters you wish to find. + +```bash +$more ~/rivanna-cli/shakespeare/Lear.text + /serpent +``` +
+ ...skipping
+     Turn all her mother's pains and benefits
+     To laughter and contempt, that she may feel
+     How sharper than a serpent's tooth it is
+     To have a thankless child! Away, away!                Exit.
+
+ +Search stops at the first occurrence. To locate the next one, type `n`. + diff --git a/content/notes/rivanna-command-line/wildcards.md b/content/notes/rivanna-command-line/wildcards.md new file mode 100644 index 00000000..e53fd2ed --- /dev/null +++ b/content/notes/rivanna-command-line/wildcards.md @@ -0,0 +1,31 @@ +--- +title: Wildcards +date: 2023-12-11-14:11:14Z +type: docs +weight: 1200 +menu: + rivanna-command-line: +--- + +**Wildcards** are special characters that can stand for strings. Wildcards enable you to work with groups of files without needing to type multiple files with similar names. + +The asterisk `*` can replace zero to unlimited characters except for a leading period. + +The question mark `?` replaces exactly one character. + +**Examples** +```bash +ls *.py +cd array_test +ls input?.dat +ls input??.dat +ls input*.dat +rm list?.sh +``` + +{{< warning >}} +BE CAREFUL when using wildcards with rm! Gone is gone! On some systems there may be backups, or there may not be, and on your personal system you would need to set up backups and learn to retrieve files. It is advisable to first run an `ls` with the pattern you plan to use with `rm`. +{{< /warning >}} + + + diff --git a/content/notes/slurm-from-cli/_index.md b/content/notes/slurm-from-cli/_index.md new file mode 100644 index 00000000..8dfa7b80 --- /dev/null +++ b/content/notes/slurm-from-cli/_index.md @@ -0,0 +1,17 @@ +--- +title: The Slurm Resource Manager +date: 2023-12-11-14:11:14Z +type: docs +weight: 100 +menu: + slurm-from-cli: +--- + +{{< figure src="/notes/slurm-from-cli/img/slurm_logo.png" width=30% >}} + + +[Slurm](https://slurm.schedmd.com/) is a __resource manager__ (RM), also known as a _queueing system_. + +Resource managers are used to submit _jobs_ on a computing cluster to compute nodes from an access point generally called a _frontend_. + +Frontends are intended for editing, compiling, and very short test runs. Production jobs go to the compute nodes through the queueing system. diff --git a/content/notes/slurm-from-cli/batch_scripts.md b/content/notes/slurm-from-cli/batch_scripts.md new file mode 100644 index 00000000..4fd27ceb --- /dev/null +++ b/content/notes/slurm-from-cli/batch_scripts.md @@ -0,0 +1,15 @@ +--- +title: Batch Scripts +date: 2023-12-11-14:11:14Z +type: docs +weight: 500 +menu: + slurm-from-cli: +--- + +Jobs are described to the resource manager in the form of a _script_. Typically this is written in the _bash_ scripting language. Bash is the default shell on most Linux-based systems, which includes the majority of HPC systems, so it is expected to be available to interpret the script. However, Slurm accepts scripts in other languages if the interpreter is available. We will consider only bash scripts in this tutorial. + +To prepare a job, the user writes a script. The top of the script is a _preamble_ that describes the resource requests. The rest of the script contains the instructions to execute the job. The script is then **submitted** to the Slurm system. The Slurm workload manager examines the preamble to determine the resources needed, while ignoring the rest of the script. It uses the resource request along with a **fair share** algorithm to set the priority of the job. The job is then placed into the requested partition to wait for the resources to become available. + +Once the job starts, the slurmd daemon runs the script as an ordinary shell script. The preamble consists of _comments_ (code that is not executed by the interpreter) so they are ignored. The rest of the script must be a valid bash shell script. + diff --git a/content/notes/slurm-from-cli/common_slurm_directives.md b/content/notes/slurm-from-cli/common_slurm_directives.md new file mode 100644 index 00000000..3397654e --- /dev/null +++ b/content/notes/slurm-from-cli/common_slurm_directives.md @@ -0,0 +1,37 @@ +--- +title: Common SLURM Directives +date: 2023-12-11-14:11:14Z +type: docs +weight: 900 +menu: + slurm-from-cli: +--- + +The most commonly used Slurm directives are listed in the table below. Many options have two versions, one with a single hyphen `-` followed by one letter, or two hyphens `--` followed by an equals sign `=` and a word. Some commands have no single-letter equivalent. + +Angle brackets `< >` indicate a value to be specified, and are not typed. + +{{< table >}} +| Single-hyphen Option | Double-Hyphen Option| Action | +| ----- | -----| ---- | +| -a \ | -\-array=\ | This is a job array with parameters in \ | +| -c \ | -\-cpus-per-task=\ | Number of cpus (cores) to be assigned to each task. For threaded code. Ensures all cores are on the same node.| +| -C \ | -\-constraint=\ | Specify certain resource constraints | +| -D \ | -\-chdir=\ | Change to \ before starting the job. Default is directory from which job is started. | +| -e \ | -\-error=\ | Separate standard error from standard output and print to file \ | +| None | -\-export=\ | Specify which environment variables are to be exported. Other options are ALL (the default) or NONE | +| | -\-gres=\ | Specify "generic consumable resources." For example `--gres=gpu:2` | +|-J \ | -\-job-name=\ | Specify a name of your choosing for the job rather than the default script name. | +| None | -\-mail-type=\ | Notify me by email upon certain events. Options are NONE (default) BEGIN (job begins), END (job ends), FAIL (job fails) , REQUEUE (job is requeued), or ALL | +| None | -\-mail-user=\ | Specify the email for notifications. | +| None | -\-mem=\ | Specify the total memory request per node, over the entire node. The default unit is megabytes. | +| None | -\-mem-per-cpu=\ | Memory request per allocated core. Default unit is megabytes. | +| -n \ | -\-ntasks=\ | Request a total number of tasks over all nodes allocated. | +| None | -\-ntasks-per-node=\ | Request that a minimum of `ntasks` be assigned to each node. +| -N \ | -\-nodes=\ | Request `nnodes` nodes. Should be used only with MPI or other protocols able to use them. | +| -o \ | -\-output=\ | Specify a name of your choosing for the standard output file rather than the default of `slurm`\`.out`. | +| -p \| -\-partition=\ | Specify the partition to run the job. | +| -t \ | -\-time=\ | Set the upper limit of the runtime. Format can be `M` (a number of minutes), `MM:SS` (minutes:seconds), `HH:MM:SS` (hours:minutes:seconds), `D-H` (days-hours), `D-HH:MM` (days-hours:minutes), or `D-HH:MM:SS` (days-hours:minutes:seconds). | +{{< /table >}} + +See also our [documentation](https://www.rc.virginia.edu/userinfo/rivanna/slurm/) for many more examples. diff --git a/content/notes/slurm-from-cli/cores_nodes_tasks.md b/content/notes/slurm-from-cli/cores_nodes_tasks.md new file mode 100644 index 00000000..84e12ab4 --- /dev/null +++ b/content/notes/slurm-from-cli/cores_nodes_tasks.md @@ -0,0 +1,30 @@ +--- +title: Cores, Nodes, and Tasks +date: 2023-12-11-14:11:14Z +type: docs +weight: 300 +menu: + slurm-from-cli: +--- + +## Hardware + +The Slurm model is a cluster consisting of a number of **nodes**. Each node is a separate server. These servers are similar to an ordinary desktop computer, but are more reliable and usually provide more memory and cores that an ordinary desktop. + +A **core** is a computing unit. It is part of a **cpu**. + +{{< alert >}} +Slurm began when cpus had only one core each. Beginning around 2005, cpus began to be divided into multiple cores. But Slurm still refers to cores as "cpus." +{{< /alert >}} + +**Memory** refers to _random-access memory_. It is not the same thing as storage. If a process reports running out of memory, it means RAM memory. Running out of disk space will result in a different error. + +For more details about the structure of a computational cluster, see our [introduction](/notes/rivanna-command-line/overview_of_rivanna/overview_terminology). + +## Processes and Tasks + +A **process** can be envisioned an instance of an executable that is running on a particular computer. Most executables run only a single process. Some executables run _threads_ within the _root_ process. + +Slurm refers to the root process as a **task**. By default, each task is assigned to one core. + + diff --git a/content/notes/slurm-from-cli/deleting_a_job.md b/content/notes/slurm-from-cli/deleting_a_job.md new file mode 100644 index 00000000..07339c5c --- /dev/null +++ b/content/notes/slurm-from-cli/deleting_a_job.md @@ -0,0 +1,34 @@ +--- +title: Deleting a Job +date: 2023-12-11-14:11:14Z +type: docs +weight: 1400 +menu: + slurm-from-cli: +--- + +## Open OnDemand + +From the Job Viewer find your jobs. If the job is pending or running, a red trash-can icon will appear under the "Actions" header. Click the icon. A dialog box will appear asking you to confirm the cancellation. + +## Command Line + +To cancel a job use the `scancel` with the job ID. You can use `squeue -u $USER` to obtain your job IDs, but you must know the JID of the specific job you wish to cancel. + +```bash +$scancel 36805 #jobID +``` + +Be aware that if a job fails due to a system failure the time will not be charged, but if you cancel your job, or it fails due to inadequate resource request, your allocation will be charged for the time expended. + +**Exercise 4** + +Write a Slurm script that requests 30 minutes of time. Submit a job that will run for at least 30 minutes. It can be some software you use; if you do not have anything set up yet, write the preamble and then add the line +```bash +sleep 30m +``` +as the command. You won't need to request a specific amount of memory. Submit this script and monitor your job's status. Once it starts, let it run for a few minutes, then cancel it. + +{{< spoiler text="Example script" >}} +{{< code-download file="/notes/slurm-from-cli/scripts/slow.slurm" lang="bash" >}} +{{< /spoiler >}} diff --git a/content/notes/slurm-from-cli/environment_variables.md b/content/notes/slurm-from-cli/environment_variables.md new file mode 100644 index 00000000..3d627f48 --- /dev/null +++ b/content/notes/slurm-from-cli/environment_variables.md @@ -0,0 +1,33 @@ +--- +title: Environment Variables +date: 2023-12-11-14:11:14Z +type: docs +weight: 1800 +menu: + slurm-from-cli: +--- +An _environment variable_ describes something about your working environment. Some of them you can set or modify; others are set by the system. To see what is currently set in your terminal, run +```bash +$printenv +``` + +To set an environment variable yourself, use the `export` command. +```bash +$export VAR=value +``` + +When your job starts, SLURM will initialize several environment variables. Many of them correspond to options you have set in your `SBATCH` preamble. Do not attempt to assign to any variable beginning with SLURM_ + +Some variables that you may wish to examine or use in your scripts: + +{{< table >}} +| Variable | Value | +| --------- | ----- | +| SLURM_SUBMIT_DIR | Directory from which the job was submitted | +| SLURM_JOB_NODELIST | List of the nodes on which the job is running | +| SLURM_JOB_ID | The numerical ID of the job | +| SLURM_NUM_TASKS | The number of tasks (obtained from --ntasks) | +| SLURM_NTASKS_PER_NODE | The number of tasks per node | +| SLURM_CPUS_PER_TASK | The number of cpus (cores) assigned to each task | +{{< /table >}} + diff --git a/content/notes/slurm-from-cli/files_and_folders.md b/content/notes/slurm-from-cli/files_and_folders.md new file mode 100644 index 00000000..438a316e --- /dev/null +++ b/content/notes/slurm-from-cli/files_and_folders.md @@ -0,0 +1,59 @@ +--- +title: Working with Files and Folders +date: 2023-12-11-14:11:14Z +type: docs +weight: 1000 +menu: + slurm-from-cli: +--- + +When using Slurm in terminal mode, you will probably want to create your own folders to organize your Slurm scripts, any input files, and the output. You will need to be able to move around from one folder to another at the terminal. + +By default, Slurm will start your job from the folder in which it was launched. You can change that with the `-D` option (directory) but many users simply navigate to the folder and type commands. + +## Creating Files and Folders + +There are several options to create, rename, and move your files and folders. Note that folders are usually called "directories" in Unix. + +### FastX + +Use the Caja file manager. This shows up as a filing-cabinet icon in the upper-left corner of the ribbon of the MATE Desktop. It can also be started from the menu Applications→System Tools→Caja. Caja's layout is very similar in appearance and behavior to Windows Explorer and similar tools. + +### Open OnDemand + +Use the File Manager to create, rename, or move your folders. + +### Command Line + +If you are familiar with the command line, you can use that. If you wish to learn it, you can go through our [Unix Tutorials for Beginners](https://learning.rc.virginia.edu/notes/unix-tutorial/), especially Tutorials 1--3. You can also go through our [HPC from the Terminal](https://learning.rc.virginia.edu/tutorials/rivanna-command-line/) tutorial if you have not already done so. + +### Changing into a Directory + +If you do not wish to learn the full command-line navigation, you will need to learn the `cd` command to get to your folder for launching your job. + +Log into a terminal in FastX, or open a terminal through the Clusters tab in Open OnDemand. + +#### Change Directory + +The `cd` command stands for "change directory." It is followed by a **path** to that directory. In the examples below, `mst3k` is a generic user ID. Substitute your own. + +```bash +$cd myworkdir +$cd /scratch/mstk3/myprojectdir +$cd +``` +The `cd` command with no options returns you to the top level of your home directory. + +You may also wish to learn `pwd` for "print working directory" so you can find out where you are. + +```bash +$cd shakespeare +$pwd +/home/mst3k/shakespeare +``` + +**Exercise 2** + +Use FastX or Open OnDemand or the command line to create a new folder under your home directory. Practice changing into and out of it. + +Use FastX and Caja to navigate to your `/scratch` directory. To get there, click `Go` in the Caja menu. A textbox will open. Be sure that "search for files" is unchecked. Erase whatever is in the textbox and type `/scratch/mst3k` (substituting your own user ID). Still in FastX, open a terminal (the black box, or in the System Tools menu) and navigate to your new scratch folder. diff --git a/content/notes/slurm-from-cli/first_script.md b/content/notes/slurm-from-cli/first_script.md new file mode 100644 index 00000000..cfd0ba8b --- /dev/null +++ b/content/notes/slurm-from-cli/first_script.md @@ -0,0 +1,19 @@ +--- +title: Our First Script +date: 2023-12-11-14:11:14Z +type: docs +weight: 700 +menu: + slurm-from-cli: +--- + +This example illustrates the main parts of a Slurm script. + +In a bash script, any text beyond the `#` is ignored. + +{{< code-download file="/notes/slurm-from-cli/scripts/hello.slurm" lang="bash" >}} + +This script runs the Python code + +{{< code-download file="/notes/slurm-from-cli/scripts/hello.py" lang="python" >}} + diff --git a/content/notes/slurm-from-cli/first_script_walkthrough.md b/content/notes/slurm-from-cli/first_script_walkthrough.md new file mode 100644 index 00000000..840e2b0e --- /dev/null +++ b/content/notes/slurm-from-cli/first_script_walkthrough.md @@ -0,0 +1,49 @@ +--- +title: The Hello.slurm Script +date: 2023-12-11-14:11:14Z +type: docs +weight: 800 +menu: + slurm-from-cli: +--- + +The first line says that this script should be run by the `bash` interpreter. +```bash +#!/bin/bash +``` + +The lines starting with `#SBATCH` are the resource requests. They are called "pseudocomments" since they have meaning to Slurm. There must be no space between `#` and `SBATCH` and the string must start in the first column of the line. + +```bash +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --mem=32000 # mb total memory +#SBATCH –-time=1-12:00:00 +#SBATCH --partition=standard +#SBATCH --account=myalloc +``` +Here we are requesting + * 1 node, 1 task + * 32GB of memory (measured in MB). Strictly speaking this will be "Gibibyes." + * 1 day and 12 hours of running time. + * The standard partition (queue). A partition must be specified. + * The account (allocation) group `rivanna-training` + +The next lines set up the environment to run our job. +```bash +module purge +module load anaconda +``` + +It is good practice to purge all modules first, since Slurm "remembers" any modules set in the environment where the script is launched. Next we load the module we need to run our program, the Python distribution Anaconda. + +Finally, we execute our job. +```bash +python hello.py +``` + +We have chosen to name this script `hello.slurm`, but it can have any name. + +**Exercise 1** + +Download the hello.slurm and hello.py scripts. Transfer them to the cluster by whatever means you wish. Modify the Slurm script to use your own allocation group name. diff --git a/content/notes/slurm-from-cli/getting_help.md b/content/notes/slurm-from-cli/getting_help.md new file mode 100644 index 00000000..36fed9bc --- /dev/null +++ b/content/notes/slurm-from-cli/getting_help.md @@ -0,0 +1,10 @@ +--- +title: Need help? +date: 2023-12-11-14:11:14Z +type: docs +weight: 4000 +menu: + slurm-from-cli +--- + +Research Computing is ready to help you learn to use our systems efficiently. You can [submit a ticket](https://www.rc.virginia.edu/form/support-request/). For in-person help, please attend one of our weekly sessions of [office hours](https://www.rc.virginia.edu/support/#office-hours). diff --git a/content/notes/slurm-from-cli/img/slurm_logo.png b/content/notes/slurm-from-cli/img/slurm_logo.png new file mode 100644 index 00000000..f3e7acd7 Binary files /dev/null and b/content/notes/slurm-from-cli/img/slurm_logo.png differ diff --git a/content/notes/slurm-from-cli/interactive_jobs.md b/content/notes/slurm-from-cli/interactive_jobs.md new file mode 100644 index 00000000..cc673bf1 --- /dev/null +++ b/content/notes/slurm-from-cli/interactive_jobs.md @@ -0,0 +1,31 @@ +--- +title: Interactive Jobs +date: 2023-12-11-14:11:14Z +type: docs +weight: 1900 +menu: + slurm-from-cli: +--- + +Most HPC sites, including UVa's, restrict the memory and time allowed to processes on the frontend. Most jobs can be submitted through the _batch_ system we have been discussing, but sometimes more interactive work is required. For example +1. Jobs that must be or are best run through a graphical interface, +2. Short development jobs, +3. "Computational steering" in which a program runs for an interval, then the output is examined and parameters may be adjusted. + +For most of these cases, we strongly recommend the use of the Open OnDemand [Interactive Applications](/notes/rivanna-intro/interactive_apps/interactive/). Jupyterlab is available to run notebooks. Rstudio and Matlab Desktop are also available to run through this interface. For more general work, including command-line options, the Desktop is usually the best option. It provides a basic terminal, but also access to other applications should they be needed. + +For general-purpose interactive work with graphics, please use the Open OnDemand Desktop. The X11 service that Linux uses for graphics is very slow over a network. Even with a fast connection between two systems, the Desktop will perform better since the X11 server process and the programs that use it are running on the same computer. + +If you must use a basic terminal for an interactive job, you must first use the command `salloc`. This is the general Slurm command to request resources. This would be followed by `srun` to launch the processes. However, this is complex and requires knowledge of the options, so we have provided a local "wrapper" script called `ijob`. + +ijob takes options similar to those used with `SBATCH`, most of which are actually arguments to `salloc`. + +```bash +$ijob –c 1 –A myalloc -t