mavis.schedule.scheduler
class mavis.schedule.scheduler.Scheduler
Class responsible for methods interacting with the scheduler
mavis.schedule.scheduler.Scheduler.__init__()
def __init__(self, concurrency_limit=None, remote_head_ssh=''):
Args
- concurrency_limit (
int
): the maximum allowed concurrent processes. Defaults to one less than the total number available - remote_head_ssh
mavis.schedule.scheduler.Scheduler.command()
Wrapper to deal with subprocess commands. If configured and not on the head node currently, will send the command through ssh
def command(self, command, shell=False):
Args
- command (
list or str
): the command can be a list or a string and is passed to the subprocess to be run - shell
Returns
str
: the content returns from stdout of the subprocess
mavis.schedule.scheduler.Scheduler.submit()
submit a job to the scheduler
def submit(self, job):
Args
- job
mavis.schedule.scheduler.Scheduler.update_info()
update the information about the job from the scheduler
def update_info(self, job):
Args
- job
mavis.schedule.scheduler.Scheduler.format_dependencies()
returns a string representing the dependency argument
def format_dependencies(self, job):
Args
- job
class mavis.schedule.scheduler.SlurmScheduler
inherits Scheduler
Class for formatting commands to match a SLURM scheduler system SLURM docs can be found here https://slurm.schedmd.com
mavis.schedule.scheduler.SlurmScheduler.submit()
runs a subprocess sbatch command
def submit(self, job):
Args
- job (
Job
): the job to be submitted
mavis.schedule.scheduler.SlurmScheduler.parse_sacct()
parses content returned from the sacct command
@classmethod
def parse_sacct(cls, content):
Args
- content (
str
): the content returned from the sacct command
mavis.schedule.scheduler.SlurmScheduler.parse_scontrol_show()
parse the content from the command: scontrol show job
@classmethod
def parse_scontrol_show(cls, content):
Args
- content (
str
): the content to be parsed
mavis.schedule.scheduler.SlurmScheduler.update_info()
Pull job information about status etc from the scheduler. Updates the input job
def update_info(self, job):
Args
- job (
Job
): the job to be updated
mavis.schedule.scheduler.SlurmScheduler.cancel()
cancel a job
def cancel(self, job, task_ident=None):
Args
- job (
Job
): the job to be cancelled - task_ident (
int
): the task id to be cancelled (instead of the entire array)
mavis.schedule.scheduler.SlurmScheduler.format_dependencies()
returns a string representing the dependency argument
def format_dependencies(self, job):
Args
- job (
Job
): the job the argument is being built for
class mavis.schedule.scheduler.SgeScheduler
inherits Scheduler
Class for managing interactions with the SGE scheduler
mavis.schedule.scheduler.SgeScheduler.parse_qacct()
parses the information produced by qacct
@classmethod
def parse_qacct(cls, content):
Args
- content (
str
): the content returned from the qacct command
mavis.schedule.scheduler.SgeScheduler.parse_qstat()
parses the qstat content into rows/dicts representing individual jobs
@classmethod
def parse_qstat(cls, content, job_id):
Args
- content (
str
): content returned from the qstat command - job_id
mavis.schedule.scheduler.SgeScheduler.submit()
runs a subprocess sbatch command
def submit(self, job):
Args
- job (
Job
): the job to be submitted
mavis.schedule.scheduler.SgeScheduler.update_info()
runs a subprocess scontrol command to get job details and add them to the current job
def update_info(self, job):
Args
- job (
Job
): the job information is being gathered for
mavis.schedule.scheduler.SgeScheduler.cancel()
cancel a job or a specific task of an array job
def cancel(self, job, task_ident=None):
Args
- job (
Job
): the job to cancel - task_ident (
int
): if specified, will cancel the given task instead of the whole array or job
mavis.schedule.scheduler.SgeScheduler.format_dependencies()
returns a string representing the dependency argument
def format_dependencies(self, job):
Args
- job
class mavis.schedule.scheduler.TorqueScheduler
inherits SgeScheduler
Class for managing interactions with the Torque scheduler
mavis.schedule.scheduler.TorqueScheduler.format_dependencies()
returns a string representing the dependency argument
def format_dependencies(self, job):
Args
- job
mavis.schedule.scheduler.TorqueScheduler.parse_qstat()
parses the qstat content into rows/dicts representing individual jobs
@classmethod
def parse_qstat(cls, content):
Args
- content (
str
): content returned from the qstat command
mavis.schedule.scheduler.TorqueScheduler.submit()
runs a subprocess qsub command
def submit(self, job):
Args
- job (
Job
): the job to be submitted
mavis.schedule.scheduler.TorqueScheduler.update_info()
runs a subprocess scontrol command to get job details and add them to the current job
def update_info(self, job):
Args
- job (
Job
): the job information is being gathered for
mavis.schedule.scheduler.TorqueScheduler.cancel()
cancel a job
def cancel(self, job, task_ident=None):
Args
- job (
Job
): the job to be cancelled - task_ident (
int
): if specified then a single task will be cancelled instead of the whole job or array
mavis.schedule.scheduler.time_format()
Converts a total seconds to a str format "H:M:S"
def time_format(total_seconds):
Args
- total_seconds
mavis.schedule.scheduler.consecutive_ranges()
Given a list of integers, return a list of ranges
def consecutive_ranges(numbers):
Args
- numbers
Examples
>>> consecutive_ranges([1, 2, 3, 4, 5, 9, 10, 14, 18])
[(1, 5), (9, 10), (14, 14), (18, 18)]