NFCORE_CUTANDRUN:CUTANDRUN:DEEPTOOLS_PLOTHEATMAP_PEAKS terminated with an error exit status (137)

[monovich]

Description of the bug

I'm getting an issue with a single file on the DEEPTOOLS_PLOTHEATMAP_PEAKS step of the pipeline (out of 140). I'm trying to resume the pipeline but it keeps hanging on this step for this file. Any advice on how to debug this? I'm trying to avoid rerunning the whole pipeline over a single file error.

Command used and terminal output

nextflow run nf-core/cutandrun \
--input ${SAMPLESHEET} \
--peakcaller 'seacr,MACS2' \
--fasta $FASTA \
--gtf $GTF \
--blacklist $BLACKLIST \
--outdir $OUTDIR \
--normalization_mode "CPM" \
--remove_mitochondrial_reads \
--mito_name "chrM" \
--dedup_target_reads \
-work-dir $WORKDIR \
-r 3.2.2

ERROR ~ Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:DEEPTOOLS_PLOTHEATMAP_PEAKS (BALL_PRIM_PED_BCRABL1_044_PSR25_CUTRUN_H3K27me3_CST9733S_native_R1)'

Caused by:
  Process `NFCORE_CUTANDRUN:CUTANDRUN:DEEPTOOLS_PLOTHEATMAP_PEAKS (BALL_PRIM_PED_BCRABL1_044_PSR25_CUTRUN_H3K27me3_CST9733S_native_R1)` terminated with an error exit status (137)


Command executed:

  plotHeatmap \
      --sortUsing sum --startLabel "Peak Start" --endLabel "Peak End" --xAxisLabel "" --regionsLabel "Peaks" \
      --matrixFile BALL_PRIM_PED_BCRABL1_044_PSR25_CUTRUN_H3K27me3_CST9733S_native_R1.computeMatrix.mat.gz \
      --outFileName BALL_PRIM_PED_BCRABL1_044_PSR25_CUTRUN_H3K27me3_CST9733S_native_R1.plotHeatmap.pdf \
      --outFileNameMatrix BALL_PRIM_PED_BCRABL1_044_PSR25_CUTRUN_H3K27me3_CST9733S_native_R1.plotHeatmap.mat.tab

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_CUTANDRUN:CUTANDRUN:DEEPTOOLS_PLOTHEATMAP_PEAKS":
      deeptools: $(plotHeatmap --version | sed -e "s/plotHeatmap //g")
  END_VERSIONS

Command exit status:
  137

Command output:
  (empty)

Command error:
  WARNING: Skipping mount /opt/singularity/4.1.3/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
  Matplotlib created a temporary config/cache directory at /tmp/matplotlib-mx3ftuav because the default path (/home/monovich/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
  .command.sh: line 6:    45 Killed                  plotHeatmap --sortUsing sum --startLabel "Peak Start" --endLabel "Peak End" --xAxisLabel "" --regionsLabel "Peaks" --matrixFile BALL_PRIM_PED_BCRABL1_044_PSR25_CUTRUN_H3K27me3_CST9733S_native_R1.computeMatrix.mat.gz --outFileName BALL_PRIM_PED_BCRABL1_044_PSR25_CUTRUN_H3K27me3_CST9733S_native_R1.plotHeatmap.pdf --outFileNameMatrix BALL_PRIM_PED_BCRABL1_044_PSR25_CUTRUN_H3K27me3_CST9733S_native_R1.plotHeatmap.mat.tab

Relevant files

nextflow.log

System information

Nextflow 24.04.1
Hardware HPC
Executor Slurm
Container Singularity
OS CentOS Linux
nf-core/cutandrun v3.2.2.-g6e1125d

[chris-cheshire]

Hi, this is not a bug persay, this is an out of memory error. Is your sample quite large? if yes then you may need to add more memory to the process with a custom config

[schweineimweltall]

Hi, I'm running the cutandrun pipeline version 3.2.2 (Nextflow 24.04.4) on my samples using

$ nextflow run nf-core/cutandrun --input samplesheet.csv --outdir /home/herbst/Documents/nextflow_EC_h3k4me3_h3k27ac --genome GRCh38 -profile docker

and get the same error message (as described above):

Caused by:
  "Process `NFCORE_CUTANDRUN:CUTANDRUN:DEEPTOOLS_PLOTHEATMAP_PEAKS (H3K27ac_R1)` terminated with an error exit status (137)"

Command executed:
  plotHeatmap \
      --sortUsing sum --startLabel "Peak Start" --endLabel "Peak End" --xAxisLabel "" --regionsLabel "Peaks" \
      --matrixFile H3K27ac_R1.computeMatrix.mat.gz \
      --outFileName H3K27ac_R1.plotHeatmap.pdf \
      --outFileNameMatrix H3K27ac_R1.plotHeatmap.mat.tab

Command error:
  Matplotlib created a temporary config/cache directory at /tmp/matplotlib-qguj_kbw because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, ...

It looks like as if my HPC with 100 GB of RAM is running out of memory and writing data to a temporary file is failing due to missing writing rights ("default path (/.config/matplotlib) is not a writable directory").

To solve the problem, I have two questions:

1. How and where do I set the MPLCONFIGDIR environment variable?

   • I tried using the command "export MPLCONFIGDIR="~/tmp/mplconfigdir" in bash. But that didn't work out.

2. How do I "add more memory to the process with a custom config" as suggested by chris-cheshire?

Kind regards,
Andreas

System information
Nextflow 24.04.4
Hardware HPC with 100 GB RAM
Container Docker
OS SLES 15.5
nf-core/cutandrun v3.2.2.-6e1125d4fe

[aakashsur]

Running into the same issue!