CAT fails on cleanup when there are .nfs files

[diekhans]

CAT can generate a failure when cleaning up when there are still .nfs files are around:

OSError: [Errno 16] Device or resource busy: '.nfs000000010cd46f5f00004178'

suggest solution is to add ignore_errors=True to the rmtree call.

Traceback (most recent call last):
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/site-packages/luigi/worker.py", line 198, in run
    new_deps = self._run_get_new_deps()
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/fieldp/bioinformatics/Comparative-Annotation-Toolkit/cat/__init__.py", line 1675, in run
    augustus_cgp(cgp_args, toil_options)
  File "/home/fieldp/bioinformatics/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 62, in augustus_cgp
    results, stdout_file_ids, param_file_id = t.restart()
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/site-packages/toil/common.py", line 882, in restart
    self._shutdownBatchSystem()
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/site-packages/toil/common.py", line 1150, in _shutdownBatchSystem
    self._batchSystem.shutdown()
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/site-packages/toil/batchSystems/singleMachine.py", line 565, in shutdown
    BatchSystemSupport.workerCleanup(self.workerCleanupInfo)
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 384, in workerCleanup
    AbstractFileStore.shutdownFileStore(workflowDir, info.workflowID)
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/site-packages/toil/fileStores/abstractFileStore.py", line 132, in shutdownFileStore
    CachingFileStore.shutdown(cacheDir)
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/site-packages/toil/fileStores/cachingFileStore.py", line 1849, in shutdown
    robust_rmtree(dir_)
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/site-packages/toil/lib/misc.py", line 55, in robust_rmtree
    shutil.rmtree(path)
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/shutil.py", line 494, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/shutil.py", line 452, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/home/fieldp/mambaforge/envs/cat3/lib/python3.7/shutil.py", line 450, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs000000010cd46f5f00004178'

[diekhans]

oh wait, this is actually being called from toil. This maybe fixed in a new version of toil