Error when running CAT with --augustus-cgp mode

[peter-d-fields]

We've recently been trying to run CAT and ran into an error when the --augustus-cgp mode is applied. Specifically, we see:

“ terminate called after throwing an instance of 'std::out_of_range'
          what():  stod
        [2023-07-02T12:27:40-0400] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:”

We used the following run parameters:

luigi \
    --module cat RunCat \
    --hal=$IN_HAL \
    --ref-genome=$REF_GENOME \
    --workers=5 \
    --config=$CONFIG \
    --local-scheduler \
    --binary-mode local \
    --maxCores=48 \
    --workDir ./tmp \
    --augustus --augustus-cgp --augustus-pb \
    --tm-cfg /home/fieldp/bioinformatics/Comparative-Annotation-Toolkit/augustus_cfgs/extrinsic.ETM1.cfg \
    --tmr-cfg /home/fieldp/bioinformatics/Comparative-Annotation-Toolkit/augustus_cfgs/extrinsic.ETM2.cfg \
    --augustus-cgp-cfg-template /home/fieldp/bioinformatics/Comparative-Annotation-Toolkit/augustus_cfgs/cgp_extrinsic_template.cfg \
    --pb-cfg /home/fieldp/bioinformatics/Comparative-Annotation-Toolkit/augustus_cfgs/extrinsic.M.RM.PB.E.W.cfg

Attempts to fix errors included:

• Restart with purged toil directory under ‘cat_work’
• Increased ‘maxCores’ and/or decreased ‘worker’ number in case ‘OSError: [Errno 16] Device or resource busy:’ was the cause of problem

Removal of ‘--augustus-cgp’ flag allows the run to complete except for errors relating to plots.

You can access the log files from the run here.

The version of AUGUSTUS in the CAT environment is:

AUGUSTUS (3.3.2) is a gene prediction tool
written by M. Stanke, O. Keller, S. König, L. Gerischer and L. Romoth.

Please let me know if any additional information would be helpful to troubleshoot this issue. Thank you for your help.

[diekhans]

Thanks for creating the ticket,

The error:
terminate called after throwing an instance of 'std::out_of_range'
what(): stod

coming from
augustus --species=human --treefile=/flashscratch/fieldp/cat/tmp/node-444a4b04-6220-4982-a758-cb87952f71b0-08842b00-fc61-4c13-ad1a-a8c328b4490a/tmp6lmzdaxa/6ee7e5cd-95de-4e8c-826e-b3763ec0f858/tmpetc94q2l.tmp --refSpecies=mm39 --referenceFile=/flashscratch/fieldp/cat/tmp/node-444a4b04-6220-4982-a758-cb87952f71b0-08842b00-fc61-4c13-ad1a-a8c328b4490a/tmp6lmzdaxa/6ee7e5cd-95de-4e8c-826e-b3763ec0f858/tmp83wljbrr.tmp --trainFeatureFile=/flashscratch/fieldp/cat/tmp/node-444a4b04-6220-4982-a758-cb87952f71b0-08842b00-fc61-4c13-ad1a-a8c328b4490a/tmp6lmzdaxa/6ee7e5cd-95de-4e8c-826e-b3763ec0f858/sumner042.63712.0003749912.tmp --param_outfile=/flashscratch/fieldp/cat/tmp/node-444a4b04-6220-4982-a758-cb87952f71b0-08842b00-fc61-4c13-ad1a-a8c328b4490a/tmp6lmzdaxa/6ee7e5cd-95de-4e8c-826e-b3763ec0f858/sumner042.63712.9062455636.tmp

is a bug in AUGUSTUS. It could be trigger by CAT giving AUGUSTUS bad input
or some other AUGUSTUS problem. To debug this, we need to get the input files
for AUGUSTUS.

CAT should have left the input files around since there was a failure.

If you can get us the input file to the failed augustus run it would help figure this out!

[peter-d-fields]

@diekhans Thank you for your reply. Are the files you need to troubleshoot the issue each of the .tmp files listed in the command? If so I'll dig them up. And do you have a preference for how to share the files with you?

[diekhans]

all the files passed to augustus, except the output file, would be what we need.

if you can put a tar.gz file of them someplace we can download it, that is the easiest. Otherwise, we can figure something else out.

[peter-d-fields]

Just a quick update. The restarted job just completed with the same error. When I went to find the files for augustus that were listed as part of the command they do not appear to exist in the tmp directory where they're expected, specifically the directory with the approximate structure:

/flashscratch/fieldp/cat/tmp/node-...

Note that set this as the tmp directory in the CAT configuration script with:

--workDir ./tmp

to deal with intermediate files creating problems on the cluster. However, when I look up the files under:

toil/augustus_cgp/jobStore/...

these files seem to exist. So it seems that maybe the when the toil tracker suggests:

[2023-07-07T19:11:40-0400] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-JobFunctionWrappingJob/instance-pyf3mo8j/file-95bf3d783176482f8b5dbc334d415fe2/$umner083.186130.7048479292.tmp' to path '/flashscratch/fieldp/cat/tmp/node-41cb2f12-cea0-40b4-9f00-166d1877ffb4-a59c56c4-f714-4137-954b-b8105410b9b1/tmpdaz2i4i_/1e269a5b-54d2-46ec-abbc-7413daeda60b/tmpl_$xep1i.tmp'

these files are not actually being moved to the ./tmp directory is expecting them? Perhaps the issue is using a relative path vs. absolute path in the configuration? Would you still like me to send the augustus inputs? I could also cp the expected inputs into the directories where they're expected and restart the pipeline?

[peter-d-fields]

Another quick update. I had a closer look at the error message in the toil log file, specifically:

       tools.pipeline.ProcException: process signaled: SIGIOT: augustus --species=human --treefile=/flashscratch/fieldp/cat/tmp/node-41cb2f12-cea0-40b4-9f00-166d1877ffb4-a59c56c4-f714-4137-954b-b8105410b9b1/tmpdaz2i4i_/1e269a5b-54d2-46ec-abbc-7413daeda60b/tmpl_6xep1i.tmp --refSpecies=mm39 --referenceFile=/flashscratch/fieldp/cat/tmp/node-41cb2f12-cea0-40b4-9f00-166d1877ffb4-a59c56c4-f714-4137-954b-b8105410b9b1/tmpdaz2i4i_/1e269a5b-54d2-46ec-abbc-7413daeda60b/tmp_xj6a1yh.tmp --trainFeatureFile=/flashscratch/fieldp/cat/tmp/node-41cb2f12-cea0-40b4-9f00-166d1877ffb4-a59c56c4-f714-4137-954b-b8105410b9b1/tmpdaz2i4i_/1e269a5b-54d2-46ec-abbc-7413daeda60b/sumner083.72978.0523157702.tmp --param_outfile=/flashscratch/fieldp/cat/tmp/node-41cb2f12-cea0-40b4-9f00-166d1877ffb4-a59c56c4-f714-4137-954b-b8105410b9b1/tmpdaz2i4i_/1e269a5b-54d2-46ec-abbc-7413daeda60b/sumner083.72978.6343653663.tmp

And tried to track down the individual files under the working directory (cat_work) to test the augustus command after placing the files in the expected locations. However, I cannot locate two of the files in the working directory at all:

sumner083.72978.0523157702.tmp
sumner083.72978.6343653663.tmp

These two files also are only found in the toil log file as part of this command, not listed anywhere else, so I'm not sure how they would be created or where to look for them. Just let me know if other information would be helpful.

[diekhans]

I believe the two missing files are output files, although --trainFeatureFile is not well documented.

You can test to see if you have all the file and get the failure by running:

augustus --species=human --treefile=tmpl_6xep1i.tmp --refSpecies=mm39 --referenceFile=tmp_xj6a1yh.tmp --trainFeatureFile=sumner083.72978.0523157702.tmp --param_outfile=sumner083.72978.6343653663.tmp

[peter-d-fields]

I see the following error:

augustus: ERROR
        Could not open the train feature file sumner083.72978.0523157702.tmp.

[diekhans]

--trainFeatureFile= is optional, so you could see if you get the failure without it.

It is a reference GFF of gene annotations used for training the model, so I suspect CAT is pulling it out of the reference GFF you are using. You could give us the files you found plus the reference GFF you have,

[peter-d-fields]

When I remove the --trainFeatureFile= option I see the following error:

augustus: ERROR
        No query file specified. Type "augustus" for help.

From the toil log I can see that the input gff is being converted to a gtf, and then his gtf is meant to be placed in the tmp directory and renamed. Specifically, from the log I see:

[2023-07-07T19:11:40-0400] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-cf7ea389267d469fa1722c7ea806b04c/Mus_musculus.GRCm39.105.chr.convert.gtf' 
to path '/flashscratch/fieldp/cat/tmp/node-41cb2f12-cea0-40b4-9f00-166d1877ffb4-a59c56c4-f714-4137-954b-b8105410b9b1/tmpdaz2i4i_/1e269a5b-54d2-46ec-abbc-7413daeda60b/tmp_xj6a1yh.tmp'

You said I should send you the set of tmp files I was able to find in the cat_work directory that relate to the augustus command?

[diekhans]

Send what you have, we will take a look and might ask for more. note, the next couple of weeks is fairly interrupted for both of us, but we will see what we can do.

[peter-d-fields]

You can use this link to download the files. Let me know if you have any questions or need any additional files.

[peter-d-fields]

@diekhans Any luck troubleshooting this issue? Any additional info I might be able to provide which would be helpful?