Feature: Optimizations and bug fixes for training (#42)

* performance optimizations

* train multiple repeats on single node in parallel

* bug fix

* fix bug in indexing when subset_samples() removed something

* sleep between jobs; stop if any job fails

* format with black

* bug fixes

* add test for MultiphenoDataloader

* update environments

* uncomment rules

* bug fixes

* subset samples in training_dataset rule

* example config.yaml

* use gpu queue for compute_burdens

* performance optimizations

* train multiple repeats on single node in parallel

* bug fix

* fix bug in indexing when subset_samples() removed something

* sleep between jobs; stop if any job fails

* format with black

* bug fixes

* add test for MultiphenoDataloader

* update environments

* uncomment rules

* bug fixes

* subset samples in training_dataset rule

* example config.yaml

* use gpu queue for compute_burdens

* refactoring

* add test for make_dataset

* update test runner for multiple conda envs

* Fix githubaction yaml

* bugfix

* fix test

---------

Co-authored-by: Brian Clarke <brian.clarke@dkfz.de>
Co-authored-by: Magnus Wahlberg <endast@gmail.com>

.github/workflows/test-runner.yml

@@ -4,6 +4,25 @@ on: [ push ]
 
 jobs:
   DeepRVAT-Tests-Runner:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v3
+      - uses: mamba-org/setup-micromamba@v1.4.3
+        with:
+          environment-name: deeprvat-preprocess-gh-action
+          environment-file: ${{ github.workspace }}/deeprvat_env_no_gpu.yml
+          cache-environment: true
+          cache-downloads: true
+
+      - name: Install DeepRVAT
+        run: pip install -e ${{ github.workspace }}
+        shell: micromamba-shell {0}
+      - name: Run pytest deeprvat
+        run: pytest -v ${{ github.workspace }}/tests/deeprvat
+        shell: micromamba-shell {0}
+
+  DeepRVAT-Tests-Runner-Preprocessing:
     runs-on: ubuntu-latest
     steps:
 
@@ -20,6 +39,6 @@ jobs:
         run: pip install -e ${{ github.workspace }}
         shell: micromamba-shell {0}
 
-      - name: Run pytest
-        run: pytest -v ${{ github.workspace }}/tests
+      - name: Run pytest preprocessing
+        run: pytest -v ${{ github.workspace }}/tests/preprocessing
         shell: micromamba-shell {0}