Merge branch 'develop' of https://github.com/Amsterdam-Music-Lab/MUSCLE…

… into develop

backend/docs/02_Start_the_application.md

@@ -0,0 +1,61 @@
+# Start the application
+
+## Before installation
+We recommend downloading a code editor, for instance [Visual Studio Code](https://code.visualstudio.com/), and to use [git](https://git-scm.com/) to copy this repository to your machine, as this makes it easier to keep your local copy up to date.
+
+## Installation with Docker
+The easiest way to run the application locally is through Docker or Podman. Docker is an application which runs a network of virtual machines ("containers") and is therefore (mostly) platform independent.
+
+The benefits of using Podman is that it is open source and can easily be used without root access. When using Podman with scripts in this documentation, `docker` and `docker-compose` should be replaced with `podman` and `podman-compose`. Alternatively, Docker could be aliased to Podman (`alias docker=podman` and `alias docker-compose=podman-compose`).
+
+### Mac OS X or Windows 10
+Install [Docker Desktop](https://docs.docker.com/desktop/).
+
+### Linux
+* Install [Docker Engine](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/), or [Docker Desktop](https://docs.docker.com/desktop/).
+* To install [Podman](https://podman.io/) on Debian-based distributions, install packages `podman` and `podman-compose`.
+
+## Development server
+Make a copy of the file .env.dist (in the same directory as this README) and rename it to .env. This file contains variables used by Docker to start up a container network serving MUSCLE.
+
+Start Docker (the app icon is a whale carrying containers). Then, open a terminal and run
+`docker-compose up` (add `sudo` on Linux).
+This command starts up the containers defined in `docker-compose.yaml`:
+- a PostgreSQL container, for storing experiment/user/playlist data, saved on the host machine in the Docker user data, represented in the volume `db_data`. Data added to the database will persist if the container is shut down.
+- a ip2country container, which provides country codes for ip addresses, used for demographic information of users.
+- a container of the server, defined in DockerfileDevelop in `backend`. The Dockerfile defines the Python version and installs development dependencies. The startup command runs migrations, adds a superuser (as defined in the `.env` file), playlist and basic experiment and then starts up a Django development server.
+- a container of the client, defined in DockerfileDevelop in `frontend`. The Dockerfile defines the node version and installs node modules. The startup command kicks off a React development server.
+
+## Navigate to the admin and user interface
+
+Once you see all containers have started up, open your browser and navigate to [http://localhost:3000](http://localhost:3000). You should now be able to see the first screen of the Goldsmiths Musical Sophistication Index questionnaire.
+
+To see the admin interface, navigate to [http://localhost:8000/admin](http://localhost:8000/admin).
+
+Since the `docker-compose.yaml` defines bind mounts for `backend` and `frontend`, any changes to the files on the host are immediately reflected in the containers, which means code watching works and hot reload works in the same way as with a native node or Django server.
+
+To stop the containers, press `ctrl-c` or (in another terminal) run
+`docker-compose down`.
+
+## Production build
+A production build should define its own `docker-compose.yaml`, making use of the `Dockerfile` of the `backend` and `frontend` environments. It should also define a custom .env file, with safe passwords for the SQL database and the Python backend. Instead of mounting the entire backend and frontend directory and using the development servers, the backend should serve with gunicorn, and the frontend should use a build script to compile static html, css and JavaScript.
+
+## Updating the application
+
+The MUSCLE platform is in continuous development and new features and bug fixes are added to the codebase on a daily basis. To update the platform, checkout the `main` branch (which is contains the latest stable release) or the `develop` branch (which contains new not yet released features), pull the latest changes using `git pull`, and rebuild the docker containers using `docker-compose up --build`. Additionally, before rebuilding the containers, you might want to check if new or updated environment variables should be updated by comparing your `.env` file with the `.env.dist` file, which contains the environment variables that can or should be used.
+
+### In short:
+
+```sh
+# 1. Checkout the main or develop branch
+git checkout main # or git checkout develop
+
+# 2. Pull the latest features, bug fixes, and other changes.
+git pull
+
+# 3. Compare your .env file with the .env.dist file and update your environment variables accordingly
+diff -y .env .env.dist # This command (available on Linux & macOS) might help you with that
+
+# 4. Rebuild & restart the containers
+docker compose up --build
+```

backend/docs/03_The_admin_interface.md

@@ -0,0 +1,19 @@
+# The admin interface
+
+After starting the containers through Docker, you can access Django's admin interface. This is used to create and change playlists, and to set up experiments.
+
+Navigate to [localhost:8000/admin](http://localhost:8000/admin) to see the admin interface.
+<img width="941" alt="admin" src="https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/40aff96c-a336-400b-8a47-4ce04d621a77">
+
+Log in:
+- username: admin
+- password: admin
+
+(This is set through the .env file. Obviously, these passwords are only suitable for local development!)
+
+You can see an overview of different Django apps:
+<img width="656" alt="adminList" src="https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/877b2850-ce62-4303-bd08-427c3c48bd46">
+- **Experiment** to add experiments and experiment series
+- **Participant** with information of participants of experiments (automatically created)
+- **Section** with information of Playlists and Sections, which may optionally be tied to Songs (i.e., artist and title of a song)
+- **Session** with information on experiment sessions (automatically created)

backend/docs/04_Creating_playlists.md

@@ -0,0 +1,65 @@
+# Creating playlists
+
+## Creating a playlist
+
+A playlist contains all the sound files used in an experiment, stored as `Section` objects. 
+The attributes of these sections can be edited in the admin interface or processed from a CSV file.
+These attributes can be used in your rules file to distinguish your sections for different purposes.
+
+Each time the playlist is saved with `Process csv` checked, the sections are updated with the attributes from the `Csv` field.
+
+#### CSV Format (comma separated):
+`artist_name [string], song_name [string], start_position [float], duration [float], "path/filename.mp3" [string], tag [string], group [string]`
+
+## Uploading sound files
+### Uploading Sound files through the admin interface:
+
+- Click `Add playlist` at `localhost:8000/admin/section/playlist/` 
+![AdminOverview](https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/80280295-1ab5-479c-b001-120a6a4f608e)
+- Give your playlist a name. Note that this name will also be used to name the folder where the sound files will be stored on the server, so make sure this name is unique.
+- Leave the `Csv` field empty, `Process Csv` unchecked and click `SAVE`. Your playlist will now appear in the list when you click on "Playlists" on the left.
+![PlaylistEmpty](https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/dd210ede-9926-479a-a636-ef8f31143ce8)
+- Click `Add sections` next to your playlist.
+![AddSection](https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/a4176951-1347-401b-9b70-a071780f065a)
+-  You can add some attributes for your sections now, or leave these blank.
+  -  When either the `artist` or the `name` field is filled a `Song` object will be created for these sections.
+  -  The `Tag` and `group` attributes can be used as identifiers in your rules file.
+![Add sections](https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/49793452/a280cb9e-15bd-40e8-8b7a-d358f60be078)
+- Click `Add sections` to upload your files to the server and add the sections to the database.
+
+Click on the name of your new playlist and notice that the `Csv` field is now filled with the attributes of the sections and the metadata of its sounds.
+
+![View playlist](https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/49793452/264679c2-ec43-4fc5-a16f-ab7841d18923)
+
+
+### Manually uploading sound files
+
+When running a local MUSCLE development installation you can just create a folder with your sound files at `backend/upload/`
+
+Running a production server you can use SSH or FTP to upload your sound files to a server, then adjust your apache / ngnx config to point to the location from which the sound files will be served.
+
+#### Generate a CSV file:
+
+- Run the `compileplaylist` command on your server container: `docker-compose run --rm server ./manage.py compileplaylist <path-relative-to-upload-folder>`.
+This will create a file named `audiofiles.csv` in the folder of your sound files.
+
+- Click `Add playlist` at `localhost:8000/admin/section/playlist/`
+![AdminOverview](https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/80280295-1ab5-479c-b001-120a6a4f608e)
+- Give your playlist a name
+- Paste the (edited) content of `audiofiles.csv` into the csv field.
+- Check `Process Csv` and click `SAVE`
+
+This will create `section` objects for your sound files with the attributes from the CSV file.
+
+## Editing the section attributes
+
+### Through the admin interface
+- Click `Edit sections` at `http://localhost:8000/admin/section/playlist/`
+![ClickEditSections](https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/4c532a39-9e1b-4325-a3a8-2485ca185a58)
+- Now you can alter the information of the sections in a table:
+![Edit sections](https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/49793452/fa21f73d-8832-457f-b26a-1c0cafc7ea6f)
+
+### Through the `Csv` field:
+- Copy the `Csv` field of your playlist in your CSV editor and make the necessary changes.
+- Paste the content of your CSV file back into the `Csv` field of your playlist.
+- Check `Process csv` and click `SAVE`

backend/docs/05_Creating_an_experiment.md

@@ -0,0 +1,24 @@
+# Creating an experiment
+
+## Admin interface
+Make sure you're logged in to the admin interface (see previous step).
+
+Then, next to "Experiments" click "Add".
+<img width="1002" alt="AddExperiment" src="https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/f71e77a1-1308-40f3-b8d5-e6f81f2ae0c0">
+
+## The Experiment form
+The form which appears lets you select the following fields:
+1. a name for your new experiment (required)
+2. a slug for the experiment (required). The slug is the part of the part of the URL which will be used to find the experiment from the frontend. In this case, the experiment will be available from `localhost:3000/someslug`.
+3. a URL with more information about the experiment (optional)
+4. a hashtag for posting about the experiment on social media (optional)
+5. a preferred language in which the experiment should be shown (optional). If this field is left "Unset", this means that the language will be determined based on the user's browser settings. Otherwise, the set language will be shown, with a fallback to English in case no translation is available.
+6. the rules of the experiment (required). This sets which logic the experiment follows. You can select implemented "rules" that have already been implemented in Python.
+7. the number of rounds the experiment should run (optional). This is set to 10 by default, which is usually fine. Not all rules files use rounds to control when specific "phases" of the experiment start or end. Think of staircasing experiments, which will present more or less difficult stimuli depending on the user's responses. These rules just ignore the "Rounds" field.
+8. bonus points (optional). Few experiments use this field, but you can use this to give bonus points to a participant who completed the experiment.
+9. a playlist (optional). For most experiments (except for those which are questionnaires) you will want to select a playlist here.
+<img width="828" alt="AddExperimentPart1" src="https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/97d57e23-aa26-40db-83ec-a1312873fdf4">
+
+10. an experiment series (optional). This is used to string several experiments together. Best to leave alone, may be deprecated.
+11. questions (optional). You can select any questions you want to ask your participants before starting an audio experiment here.
+<img width="992" alt="AddExperimentPart2" src="https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/d8253af5-9be2-4428-a90a-9e612c863fb8">

backend/docs/06_Custom_questions.md

@@ -0,0 +1,54 @@
+# Custom questions
+
+## The backend directory
+To add questions that haven't yet been added to the MUSCLE infrastructure, you'll have to navigate to the backend directory. It has the following file structure:
+```bash
+.
+├── aml
+├── experiment
+├── locale
+├── logs
+├── participant
+├── requirements
+├── requirements.in
+├── result
+├── section
+├── session
+└── upload
+```
+
+The backend is written in Python, so a little bit of familiarity with programming is required. Within the backend directory, go to the `experiment` directory. Within, you find the following file structure:
+```bash
+.
+├── actions
+├── fixtures
+├── management
+├── migrations
+├── questions
+├── rules
+├── standards
+├── templates
+└── tests
+```
+
+The files in the `rules` directory specify the logic of experiments. We'll turn to those later.
+
+The files in the `questions` directory contain the questions, as Python classes. For instance, if you look into the file `goldsmiths.py` you'll see the following:
+<img width="720" alt="Screenshot 2023-11-14 at 14 30 51" src="https://github.com/Amsterdam-Music-Lab/MUSCLE/assets/11174072/723f1add-602c-4e34-ae01-4b683af56157">
+
+We have different question types available, all imported from `backend/experiment/actions/forms.py`:
+- NumberQuestion (number field)
+- TextQuestion (text field)
+- BooleanQuestion (yes/no buttons)
+- ButtonArrayQuestion (more than 2 buttons)
+- RadiosQuestion (radio buttons)
+- DropdownQuestion (dropdown menu)
+- AutoCompleteQuestion (dropdown menu with autocomplete)
+- RangeQuestion (slider)
+- LikertQuestion (slider in which chosen category is displayed on top)
+
+As you can see, the Goldsmith's Musical Sophistication Index uses many LikertQuestions. LikertQuestions can be initialized with a `key` and a `question` argument. The key helps us find the responses to the question back in the database, so it's a good idea to use a unique, recognizable key. The key should not contain other characters than numbers, letters and underscores.
+
+The `question` is the prompt shown to a participant. Here, it is wrapped in a translation tag (represented by `_( )`): this means the question can be translated into different languages.
+
+If you wish to add questions to the infrastructure, you are of course welcome to so in a fork of the project. However, if you think the questions may be of interest to other users, consider making a [git branch](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging) and contribute to this repository with a [pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request).

backend/docs/07_Actions_overview.md

@@ -0,0 +1,55 @@
+# Overview of the different actions for experiments
+
+The actions that can be used by experiments are formalized in `backend/experiment/actions`
+
+## 1. Explainers
+An Explainer presents some information to participants of an experiment. They can include a number of Steps, which may or may not be numbered. 
+[Preview](https://amsterdam-music-lab.github.io/MUSCLE/?path=/story/explainer--default)
+
+## 2. Playlist
+If there are multiple playlist tied to an experiment, the Playlist widget lets participants select one of them for use in the experiment.
+
+## 3. Consent
+This action is used to present a consent form to the user, and a button to agree, or quit the experiment.  
+The content of the consent agreement can be loaded from the rules file or uploaded from the admin interface.
+
+A consent form uploaded from the admin interface will override the one loaded from the rules file, so that each instance of an experiment can have its own consent file.  
+Click on your experiment in the admin interface to upload your custom consent file at ```localhost:8000/admin/experiment```
+
+Files loaded from the rules file must be placed in ```admin/experiment/templates/consent```  
+Files uploaded from the admin interface are placed in ```backend/upload/consent```
+
+Allowed file formats:
+- HTML (.html)  
+    - May contain django template tags
+- MARKDOWN (.md)
+
+### Initializing a Consent form
+```
+consent = Consent(
+            experiment.consent,
+            title=_('Informed consent'),
+            confirm=_('I agree'),
+            deny=_('Stop'),
+            url='consent/<template file>'
+            )
+```
+
+[Click here for an online editor for markdown files - (dillenger.io)](https://dillinger.io/)
+
+## 4. Redirect
+This action allows to redirect to another website.
+
+## 5. Score
+This action shows scores or messages to participants after trials.
+
+## 6. Final
+This action finishes the session for the participant, and shows a final score or message.
+
+## 7. Trial
+This action presents questions, audio and text to a user, and will be usually repeated multiple times ("rounds") in an experiment. It may contain any of the following elements:
+- a Form object with one or multiple Questions
+- a Playback object describing different player widgets
+- a Html element to present any other kind of information, such as images or videos, to the participant
+
+The next sections will give an overview of Form and Playback objects.

backend/docs/08_Forms_and_question_widgets.md

@@ -0,0 +1,35 @@
+# Forms and question widgets
+
+As mentioned before, a `Trial` may contain a `Form`, which present one or more questions to the user. We have a choice of multiple question widgets:
+
+- [Autocomplete](https://amsterdam-music-lab.github.io/MUSCLE/?path=/story/question--autocomplete)  
+- [ButtonArray](https://amsterdam-music-lab.github.io/MUSCLE/?path=/story/question--button-array)  
+- [Checkboxes](https://amsterdam-music-lab.github.io/MUSCLE/?path=/story/question--checkboxes)  
+- [Dropdown](https://amsterdam-music-lab.github.io/MUSCLE/?path=/story/question--dropdown)  
+- [Radios](https://amsterdam-music-lab.github.io/MUSCLE/?path=/story/question--radios)  
+- [TextRange](https://amsterdam-music-lab.github.io/MUSCLE/?path=/story/question--text-range)  
+- [IconRange](https://amsterdam-music-lab.github.io/MUSCLE/?path=/story/question--icon-range)  
+- [String](https://amsterdam-music-lab.github.io/MUSCLE/?path=/story/question--default)  
+
+## Initializing a Question
+An example of how a question is initialized in the Python backend:
+```python
+question = RadiosQuestion(
+        key=key,
+        question=_('Does this sound like song or speech to you?'),
+        choices=[
+            _('sounds exactly like speech'),
+            _('sounds somewhat like speech'),
+            _('sounds neither like speech nor like song'),
+            _('sounds somewhat like song'),
+            _('sounds exactly like song')],
+        result_id=prepare_result(key, session, section=section, scoring_rule='LIKERT')
+    )
+```
+The `key` is a short name of the question. With every question, we also send a `result_id`. This means we can already register information in the database, such as (in this case) the section this question refers to, or the scoring rule by which we assign a score after the participant gives a response.
+
+## Initializing a Form
+After defining the question above, we can then wrap it in a form as follows:
+```python
+form = Form([question])
+```