error in INPUT_CHECK:COMBINE_SCOREFILES

[scienception]

Description of the bug

I was following the tutorial and I got the same error when running both on Mac M2 and Linux, Conda and Docker.

Command used and terminal output

nextflow run pgscatalog/pgsc_calc -profile test,conda
nextflow run pgscatalog/pgsc_calc -profile test,docker

Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Error executing process > 'PGSCATALOG_PGSCALC:PGSCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)'

Caused by:
  Process `PGSCATALOG_PGSCALC:PGSCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)` terminated with an error exit status (127)

Command executed:

  combine_scorefiles -s PGS001229_22.txt             -t GRCh37             -o scorefiles.txt.gz             -l log_scorefiles.json             -v

  cat <<-END_VERSIONS > versions.yml
  COMBINE_SCOREFILES:
      pgscatalog_utils: $(echo $(python -c 'import pgscatalog_utils; print(pgscatalog_utils.__version__)'))
  END_VERSIONS

Command exit status:
  127

Command output:
  (empty)

Command error:
  .command.sh: line 2: combine_scorefiles: command not found

Work dir:
  /users/mcevallo/work/a7/f9437a3442f4d61c60f879fa7edd12

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
ERROR ~ ERROR: No scores calculated!

 -- Check '.nextflow.log' file for details

Any idea of what may be going on?

### Relevant files

_No response_

### System information

I'm using Java 20.0.1. conda 23.3.1, docker 24.0.2

[nebfield]

Sorry, I'm unable to reproduce your problem. Following the tutorial on Linux:

$ nextflow run pgscatalog/pgsc_calc -profile test,singularity

Runs v1.3.2 to completion. On an M1 Mac:

$ nextflow run pgscatalog/pgsc_calc -profile test,docker --platform arm64

Works as well.

Some things to try:

• Set the explicit version:
  • nextflow run pgscatalog/pgsc_calc -r v1.3.2 -profile test,docker --platform arm64
• If you ran an old version of pgsc_calc before:
  • Your nextflow cache may be invalid: nextflow run pgscatalog/pgsc_calc -r v1.3.2 -latest -profile test,docker --platform arm64
• Perhaps nextflow self-update will help

[scienception]

I guess I kind of got it working with the --platform arm64 flag. But it doesn't go as smoothly as the one shown in the tutorial. It seems to skip several things:

------------------------------------------------------
executor >  local (3)
executor >  local (4)
executor >  local (5)
executor >  local (6)
executor >  local (7)
executor >  local (8)
executor >  local (8)
executor >  local (8)
[06/0aa25e] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1 ✔
[19/e0605f] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1 ✔
[skipped  ] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1, stored: 1 ✔
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[ed/171ede] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1 ✔
[b0/4036b0] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1 ✔
[41/83dcf4] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1 ✔
[37/f1952b] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1 ✔
[1a/c17016] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1 ✔
[f2/5d5b26] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1 ✔
[skipping] Stored process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELBIM (1)
INFO: Scores ready for calculation

[scienception]

How can I take a look at the minimal dataset being used here?

[nebfield]

Some of the processes were skipped because they completed successfully the first time you ran the workflow, before you hit the combine scorefiles error. We try to cache work to save time (and CPU hours/electricity 😁 ).

If you'd like to run without using a cache you could delete the work directory (rm -r work/) and retry.

The test genomes are here:

https://github.com/PGScatalog/pgsc_calc/tree/main/assets/examples/target_genomes

It's a subset of variants reused from the CINECA project.

[scienception]

I just have one more question. When running directly nextflow run pgscatalog/pgsc_calc -profile singularity --input ../samplesheet.csv --target_build GRCh37 --scorefile ../scorefiles/scorefile.txt it works fine. But then I tried:

nextflow run pgscatalog/pgsc_calc -profile singularity --input ../samplesheet.csv --target_build GRCh37 --pgs_id PGS001229 --chrom 22
And it gives this error:

Command error:
  root: 2023-07-12 00:33:38 DEBUG    Verbose logging enabled
  pgscatalog_utils.config: 2023-07-12 00:33:38 DEBUG    Using 2 threads to read CSVs
  pgscatalog_utils.config: 2023-07-12 00:33:38 DEBUG    polars threadpool size: 2
  pgscatalog_utils.match.read: 2023-07-12 00:33:38 DEBUG    Reading scorefile
  pgscatalog_utils.match.read: 2023-07-12 00:33:38 DEBUG    --chrom parameter not set, using all variants in scoring file
  pgscatalog_utils.match.preprocess: 2023-07-12 00:33:38 DEBUG    Complementing column effect_allele
  pgscatalog_utils.match.preprocess: 2023-07-12 00:33:38 DEBUG    Complementing column other_allele
  pgscatalog_utils.match.combine_matches: 2023-07-12 00:33:38 DEBUG    Reading matches
  pgscatalog_utils.match.combine_matches: 2023-07-12 00:33:38 DEBUG    Labelling match candidates
  pgscatalog_utils.match.label: 2023-07-12 00:33:38 DEBUG    Labelling best match type (refalt > altref > ...)
  pgscatalog_utils.match.label: 2023-07-12 00:33:38 DEBUG    Labelling duplicated best match: keeping first instance as best_match = True
  pgscatalog_utils.match.label: 2023-07-12 00:33:38 DEBUG    Labelling multiple scoring file lines (accession/row_nr) that best_match to the same variant
  pgscatalog_utils.match.label: 2023-07-12 00:33:38 DEBUG    Labelling all duplicates with exclude flag
  pgscatalog_utils.match.label: 2023-07-12 00:33:38 DEBUG    Labelling ambiguous variants
  pgscatalog_utils.match.preprocess: 2023-07-12 00:33:38 DEBUG    Complementing column REF
  pgscatalog_utils.match.label: 2023-07-12 00:33:38 DEBUG    Labelling ambiguous variants with exclude flag
  pgscatalog_utils.match.label: 2023-07-12 00:33:38 DEBUG    Labelling multiallelic matches with exclude flag
  pgscatalog_utils.match.label: 2023-07-12 00:33:38 DEBUG    Not excluding flipped matches
  pgscatalog_utils.match.filter: 2023-07-12 00:33:38 DEBUG    Filtering to best_match variants (with exclude flag = False)
  pgscatalog_utils.match.filter: 2023-07-12 00:33:38 DEBUG    Calculating overlap between target genome and scoring file
  pgscatalog_utils.match.filter: 2023-07-12 00:33:39 ERROR    Score PGS001229_hmPOS_GRCh37 fails minimum matching threshold (1.52% variants match)
  pgscatalog_utils.match.match_variants: 2023-07-12 00:33:39 CRITICAL Error: no target variants match any variants in scoring files
  Traceback (most recent call last):
    File "/venv/bin/combine_matches", line 8, in <module>
      sys.exit(combine_matches())
    File "/venv/lib/python3.10/site-packages/pgscatalog_utils/match/combine_matches.py", line 36, in combine_matches
      log_and_write(matches=matches, scorefile=scorefile, dataset=dataset, args=args)
    File "/venv/lib/python3.10/site-packages/pgscatalog_utils/match/match_variants.py", line 90, in log_and_write
      raise Exception("No valid matches found")
  Exception: No valid matches found

Why is the second one not working?
I used your test files and sample sheet.

[nebfield]

• PGS001229 contains 51,209 variants across many chromosomes
• The synthetic test genomes have very few variants, all on chromosome 22 (800ish?)
• The test scoring file was made from PGS001229 but isn't the same score as the one published in PGS Catalog (it contains a small subset of variants)

The test profile is used in our automated tests to make sure the workflow runs correctly and to verify pgsc_calc is installed properly. The test data isn't biologically meaningful and isn't good for testing against scores published in the PGS Catalog.

The synthetic HAPNEST genomes are probably better for testing larger scores, but you'll have the best chance of success with real genomes that have been imputed.

There's stringent quality control to check the overlap between a scoring file and a target genome. If the variants don't overlap then pgsc_calc is not reproducing the score published in the PGS Catalog accurately (which is bad 😅 ). We don't natively support calculating scores using a subset of chromosomes (i.e. --chrom doesn't exist/work) because of this. All parameters are available in the docs.