colrc-v2

Coeur d'Alene Online Language Resource Center Version 2.0

Table of Contents

• Getting Started
  • Requirements
  • First Installation
  • Subsequent Pulls
  • Cleanup
• Working with Git
  • The GitHub Workflow
    • Checkout
    • Add
    • Commit
    • Push
  • Working in this Repository
• About
  • Frontend
  • Backend
  • Postgres
  • Testing
    • Frontend
    • Backend

Getting Started

Our recommended development environment can be easily launched using docker-compose. Note that when we refer to 'the command line,' we mean the WSL/Linux/Mac command line (not Windows Powershell, nor the Windows-native terminal app).

Here are the steps we recommend to start (as of 2/28/2023). If this is your first time using the development environment, read the requirements below and the first installation steps. If you are reinstalling or trying to integrate new changes, see the subsequent installation steps.

Requirements

Install these applications in the appropriate format for your machine (Windows, Mac or Linux). Windows users should ensure that they are using the 'education' or 'enterprise' editions of WindowsOS, the 'home' editions do not fully support WSL.

• You will need sudo/root access on your system at the command line.
• docker
• docker-compose
• git
• python3
• node
• If you are running Windows, you'll need to use WSL with Debian. Once you have Debian installed, you will need to run sudo apt-get update and then sudo apt-get upgrade. You will also want to be sure you configure Docker Desktop to connect with WSL
• We recommend using VSCode as your code editor for this project.

Note that if you're using one of our Jetstream2 virtual machines, all of these prereqs are already there except for node. You'll only need node and NPM if you're working on running or trouble-shooting node processes outside of the docker container. But they're very easy to install on Jetstream - you can just type node or npm at the command line, and that system will tell you the command to use to install these libraries.

First Installation

You will do these steps at a command line - windows users will use the WSL/Debian command lime; Mac and Linux users will use your regular terminal application.

0. Set up an ssh key to work with your github account:

First, follow the steps to generate a new SSH Key, and then add the key to your github account

1. Create and/or switch to the directory where you want our application to live. Then, from the command line in that directory, clone this repository:

   git clone git@github.com:arizona-linguistics/colrc-v2.git

2. Afterward, change to the newly-created directory and pull to make sure you have all of the current changes to the repository. Note that the default branch, "main," is the branch you should clone and/or pull.

   cd colrc-v2 && git pull

3. Create the external directories needed to run Odinson: Move up one directory, cd .., and create a new directory named 'data' mkdir data.
   Move into the new directory, cd data, and make another directory called 'odinson', mkdir odinson.

Move back to the root of the colrc-v2 directory, cd .. to move up one level, then cd colrc-v2 to move into the root of the colrc-v2 dirctory.

4. Create your docker override yml file. The purpose of this file is to tell our application where your external Odinson directory lives, so that it can map a drive to that directory. The override file will not be copied back into the public git repo, as it will have information in it that's specific to you.

You will make a new file at the command line: touch docker-compose.override.yml, OR open your VS Code editor from the command line by typing code .. This command will open VSCode in the directory where you are currently working, so you'll be able to see all the files there and manipulate them. If you open VSCode, you can right or control-click on the Explorer view and select 'new text file', and save the new file as docker-compose.override.yml.

No matter how you created the file, copy and paste the following code block into the file. Then change the [yourfilepath] to match your path to the data/odinson directory. If you're not sure what yourfilepath is, you can type pwd (print working directory) at the command line to reveal your current location.

services:
  odinson-rest-api:
    volumes: 
      - /[yourfilepath]/data/odinson:/data/odinson

If you are on a Jetstream2 VM, and if you created a src directory within your home dir to use as your coding homebase, your docker-compose.override.yml file will be:

services:
  odinson-rest-api:
    volumes: 
      - /home/exouser/src/data/odinson:/data/odinson

If you are working on a Mac with an Apple Silicon chip, you may need to specify the platform for your Hasura image in your docker-compose.override.yml file as well. If so, your file would look like this:

services:
  odinson-rest-api:
    volumes: 
      - [yourfilepath]/data/odinson:/data/odinson
  hasura:
    platform: linux/amd64

5. Next, download our image, pdf and audio files from Dropbox. As files are updated in our Dropbox folder, you can run the script below while the development environment is down to keep your local filesystem up to date. From the colrc-v2 directory:

   cd ./misc and then python3 dropbox-sync.py

If you do not have the requests library, you will need to install it:

  `pip3 install requests` or `python3 -m pip install requests`

If you get a permissions error when running the script, you can use the command below to fix the permissions you need (if the file/folder that has the permissions issue is different, substitute file_data with that file/folder):

  `sudo chown -R $USER file_data`

6. Only if you are running the development environment on a VM, you will need to change the clients in frontend/src/App.js. Nano or vi that file, find the functions that set up the two clients we use, these are defined immediately below the import statements. Each will have a uri that uses localhost. Change localhost to the IP address of your hosted VM. In each case

      new HttpLink({
      uri: 'http://localhost:8080/v1/graphql',
      //uri: process.env.REACT_APP_AUTH_CLIENT
    })
   

   will become

      new HttpLink({
      uri: 'http://[yourIPaddress]:8080/v1/graphql',
      //uri: process.env.REACT_APP_AUTH_CLIENT
    })
   

7. At the command line, build our development environment. Depending on your configuration, you may or may not need to sudo The initial build may take a while, but subsequent builds will go faster.

   docker-compose -f docker-compose.yml -f docker-compose.override.yml up --build or docker compose -f docker-compose.yml -f docker-compose.override.yml up --build

The environment is fully up and running when you see a message that says 'Compiled successfully!'. At that point, you can go to http://localhost:3000 if you're working on your own machine, or http://[yourIPaddress]:3000 if you're working on a virtual machine, and you'll be able to access the running application! The terminal window that you've used for this command will be occupied - it will not come back to a command prompt. This is because the process is not daemonized. If you want to run the application in a deamonized way, you can add a -d flag after up.

9. If you are working on a remote VM, you'll need to open the hasura console by directing your browser to [yourIPaddress]:8080/console. You can find the required admin secret in the docker-compose.yml (hasura_graphql_admin_secret) file at the project root. Navigate to the data tab, and On each of these four tables: audiofiles, textfiles, textimages, elicitationfiles, under modify, you'll find a computed field. Edit the computed field - which is an SQL statement that tells the machine where it can find our pdf, png, wav and mp3 files. In the SQL statement is a URL path. Change the stem of that path (which is probably localhost) to [yourIPaddress]:80. Then click the button to execute the SQL.

10. When you want to bring the system down, you can either use control-C from the terminal where the application is running; or use the 'down' button to the right of the container in Docker Desktop's gui, or you can open a new terminal, navigate to the root of the project, and use this command:

`docker-compose down`

To relaunch for a new work session, if you haven't done a new pull from the repo, you can just 'up' the system without rebuilding it like this:

`docker-compose -f docker-compose.yml -f docker-compose.override.yml up`

To relaunch after a new pull or significant local changes to i.e. the backend, you can build and then up like this:

`docker-compose -f docker-compose.yml -f docker-compose.override.yml up --build`

Subsequent Pulls

As we progress in development, this repository will change. To get the most recent version of the repository, you will need to pull from the main branch. You will need to take a look at the most recent commits to see whether there have been changes to colrc.sql (which is the file that defines the database, including table permissions and relations via Hasura). Then:

1. Make sure that the development environment is currently not running:

   docker-compose down

2. If there have been changes to colrc.sql since your last pull, delete the database's data folder:

   sudo rm -rf misc/db_data

3. Pull the changes from the main branch:

   git checkout main && git pull

4. Follow the first installation steps starting from step 3.

Cleanup

To easily remove old volumes and containers, you can run ./cleanup.sh from the base directory.

Working with Git

This section will serve as a tutorial on how to use Git/GitHub, as well as a guide regarding our specific workflow in this repository.

The GitHub Workflow

In short, the basic GitHub workflow is checkout > add > commit > push, and work is done on a branch before it is merged into the main part of the repo. This workflow is described more in detail below. As you begin to develop code, please make sure that you create and checkout a git branch that is named for the issue that you're working on. We use the issue number as at least part of the branch name so that we can better track which branches address which issues.

Checkout

• git checkout <branch> switches your branch to the one you specified, and git checkout -b <new-branch> <starting-point> will make a new branch and switch to it, using another branch as a starting point (which is typically the main branch).

• Branches allow us to make changes separately and merge them when we are ready, rather than having your code constantly be changed by someone else from underneath you.

Add

• After you have changed a file, you can run git add <file> to add a file to your future commit (these files are now in what is called the staging area). To remove a file from the staging area, you may run git reset <file>.
  • A commit contains all of the changes that you (or others) add, and is like a snapshot of the repository's state at that time. This allows us to keep track of changes that are made in an efficient manner.

Commit

• Once you have made all of the changes that you want to make for a particular issue/subset of issues, you can run git commit -m "<message>" to make a commit for these changes.

• To see what a commit/commit message should look like, take a look at previous commits in this repository. Your changes in a commit should be somewhat related to each other, so that way your message is succinct and changes are easier to track.

Push

• Once you are ready to send your commit(s) to the GitHub repository, you may run git push origin <branch> to publish your changes.
  • Note that while you may reverse your changes on your local branch if you make a mistake (for example, git reset HEAD~1 will undo your most recent commit), it is very frowned upon to do this after your changes have been already pushed. Before you push, make sure you have made any corrections that you want to make first.

If you change the database (tables or permissions)

• Produce a dump of the database with Hasura metadata included.

• Change directory to the root of the repo.

• Start the container if it's not already running, dump the database, and shut it back down.

  sudo docker-compose up -d

  cd <rootOfRepo>

  cd colrc-v2/misc/sql && pg_dump -Cc -U postgres -h localhost colrc > colrc_new.sql

  Password: (get the pwd from the team)

  sudo docker-compose down

• Open colrc_new.sql and

• Change "DROP DATABASE colrc" to

  DROP DATABASE IF EXISTS colrc

• If you are using pg_dump v13+, also change "LOCALE = 'en_US.utf8';" to

  LC_COLLATE = 'en_US.utf8' LC_CTYPE = 'en_US.utf8';

• Once you have verified that all looks well in the dump, you can overwrite the previous

  mv colrc.sql colrc_old.sql

  mv colrc_new.sql colrc.sql

• Then commit your changes and push to the repo.

Working in this Repository

In order to address an issue with the code (which may be an unimplemented feature, a bug, or something which may need to be rewritten), first check the issues page of the repository. Here you will find a list of issues that you can work on.

For example, say you choose the issue below:

Here is how you would go about addressing this issue:

1. Open the issue and look at the instructions. Once you have finished reading them, make sure you have the latest changes to the repository:

   git pull

2. Then, make a new branch for the issue based off of our main branch:

   git checkout -b issue141 main

3. For every set of changes, add and commit them with a message explaining what you changed--for example:

   git add frontend/pages/AffixTable.js frontend/pages/AudioTable.js

   git commit -m "Added comments for documentation"

4. Once you are ready and satisfied with your changes, push them to GitHub:

   For the first push: git push -u origin issue141

   Subsequent pushes: git push origin issue141

5. When you have completed the issue and you think the changes are ready to be implemented, go to the pull requests page and submit a new pull request from your branch to the main branch.

   When you make a pull request, it is important to explain the following:

   • What changes you made
   • What issue your changes fixed
   • How your changes were tested

   Below is a sample pull request that includes these elements:

   

About

The following sections discuss the components we are using for this project, along with ways you can access them.

Frontend

The frontend is built using react. If you've launched the development environment using docker-compose, the app will launch on http://localhost:3000. Changes you make to the frontend source will update in realtime.

After making changes to frontend files, please manually run prettier to ensure that all code follows the same standards of style. To do this, after making any changes, run the command 'npx prettier [file name or file spec e.g. /home/exouser/colrc-v2/frontend/src/pages/TextTable.js or */*Table.js] --write'.

Backend

The backend is a Node app that currently relies on Express, sequelize, Hasura GraphQL, and Postgres. If you've launched the development environment using docker-compose, any changes you make to the backend source are monitored with nodemon, and will trigger a rebuild whenever detected.

You can access the Hasura GraphQL Console by going to http://localhost:8080 in your browser.

Postgres

Postgres may be accessed directly by running:

docker exec -it "colrc-v2-postgres-db" psql -d colrc -U root -W

See .env for the credentials used to launch the development version of the service.

Place .sql files in misc/sql to have them loaded when Postgres (v12) first launches.

Testing

We suggest testing using the environment launched by docker-compose. Both frontend and backend tests are written using jest:

Frontend

docker exec -it "colrc-v2-frontend" npm run test

Backend

docker exec -it "colrc-v2-backend" npm run test