This repository has been archived by the owner on Sep 23, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 4
Add queue interaction support #21
Merged
Merged
Changes from 23 commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
645fff5
Modify the command to accept the parameters needed for cluster enviro…
josenavas 8cc9938
Adding cluster utils
josenavas 6f891ff
Adding PBS submission support to the bench suite
josenavas db36cca
Modify optparse interface to accept the new parameters for PBS support
josenavas 8e1c872
Fixing file name typo
josenavas ecc7952
Fixing tests
josenavas 115ce33
Making the prefix of each command unique
josenavas 4de21c2
Add job id tracking on the shell script
josenavas 70ab5a9
Support multiple bash variables for keeping track of job_id and creat…
josenavas 8babbda
Modifying BenchResultsProcesser command to accept a list of job ids t…
josenavas 4aaa353
Modifying interface to accept the new parameter
josenavas d43128f
Fixing typo
josenavas 8e0a680
Adding wait_on functionality
josenavas f1683fc
adding tests for the commands
josenavas a69bc0c
Fixes issue #18
josenavas ca7f12d
Fixing typo
josenavas a091210
Fixing subprocess communication
josenavas 1dab36e
Fixing typo
josenavas d0a9097
Fixing cluster utils
josenavas 30c50b4
Fixing cluster utils
3498886
Fixing @antgonza comments
josenavas 49455b8
Merge branch 'pbs' of github.com:josenavas/QIIME-Scaling into pbs
josenavas 5835c80
Addressing PEP8 issues and improving documentation
josenavas 5dbbf3f
Fixing imports
josenavas ff0dd82
Fixing default value
josenavas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
#!/usr/bin/env python | ||
|
||
__author__ = "Jose Antonio Navas Molina" | ||
__copyright__ = "Copyright 2014, The QIIME-Scaling Project" | ||
__credits__ = ["Jose Antonio Navas Molina", "Daniel McDonald"] | ||
__license__ = "BSD" | ||
__version__ = "0.0.1-dev" | ||
__maintainer__ = "Jose Antonio Navas Molina" | ||
__email__ = "josenavasmolina@gmail.com" | ||
__status__ = "Development" | ||
|
||
from subprocess import Popen | ||
from os import environ | ||
from time import sleep | ||
|
||
|
||
def check_status(jobs_to_monitor): | ||
"""Check the status of the passed list of jobs | ||
|
||
Inputs: | ||
jobs_to_monitor: list of job ids | ||
|
||
Returns: | ||
A subset of jobs_to_monitor containing those jobs that are still | ||
running | ||
""" | ||
# Get all the commands running pf the current user | ||
user = environ['USER'] | ||
qstat_cmd = "qstat | grep %s" % user | ||
proc = Popen(qstat_cmd, stdout=subprocess.PIPE, | ||
stderr=subprocess.PIPE, shell=True) | ||
(stdout, stderr) = proc.communicate() | ||
# Parse the qstat output | ||
lines = stdout.splitlines() | ||
running_jobs = [] | ||
for l in lines: | ||
job_id, job_name, user, time, status, queue = l.split() | ||
job_id = job_id.split('.')[0] | ||
# Check if this job is one of the jobs that we have to | ||
# monitor and check if it is running or queued | ||
if job_id in jobs_to_monitor and status in ['R', 'Q']: | ||
running_jobs.append(job_id) | ||
# Return the list with the running jobs that we're still waiting for | ||
return running_jobs | ||
|
||
|
||
def wait_on(jobs_to_monitor, poll_interval=5): | ||
"""Block while jobs to monitor are running | ||
|
||
Inputs: | ||
jobs_to_monitor: list of job ids | ||
poll_interval: interval between checks, in seconds | ||
""" | ||
# Get the jobs ids by up to the first '.' character | ||
jobs_to_monitor = [job.split('.')[0] for job in jobs_to_monitor] | ||
# Loop until there is some job to monitor | ||
while jobs_to_monitor: | ||
# Sleep before new job status check | ||
sleep(poll_interval) | ||
# Check job status and get new set of jobs to wait on | ||
jobs_to_monitor = check_status(jobs_to_monitor) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will it be possible to ignore empty lines and those that start with # (comments).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That should never happen. Note that lines is the output of
qstat | grep user
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wrong line, meant to add it to were the parsing of the text file to create jobs happens.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The creation of jobs happens in bash, so I don't really know where you meant 😖
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it, my bad. Was confused with the way that cluster_jobs.py works.