scheduler module

class mavis.schedule.scheduler.Scheduler(concurrency_limit=None, remote_head_ssh='')

Bases: object

Class responsible for methods interacting with the scheduler

Parameters:concurrency_limit (int) – the maximum allowed concurrent processes. Defaults to one less than the total number available

ENV_JOB_IDENT = '{JOB_IDENT}'

the expected pattern of environment variables which store the job id

Type:str

ENV_TASK_IDENT = '{TASK_IDENT}'

the expected pattern of environment variables which store the task id

Type:str

HEADER_PREFIX = '#'

cancel(job, task_ident=None)

command(command, shell=False)

Wrapper to deal with subprocess commands. If configured and not on the head node currently, will send the command through ssh

Parameters:command (list or str) – the command can be a list or a string and is passed to the subprocess to be run Returns:the content returns from stdout of the subprocess Return type:str

format_dependencies(job)

returns a string representing the dependency argument

submit(job)

submit a job to the scheduler

update_info(job)

update the information about the job from the scheduler

wait()

class mavis.schedule.scheduler.SgeScheduler(concurrency_limit=None, remote_head_ssh='')

Bases: mavis.schedule.scheduler.Scheduler

Class for managing interactions with the SGE scheduler

Parameters:concurrency_limit (int) – the maximum allowed concurrent processes. Defaults to one less than the total number available

ENV_ARRAY_IDENT = 'JOB_ID'

ENV_JOB_IDENT = 'JOB_ID'

expected pattern for environment variables which store the job id

Type:str

ENV_JOB_NAME = 'JOB_NAME'

expected pattern for environment variables which store the job name

Type:str

ENV_TASK_IDENT = 'SGE_TASK_ID'

HEADER_PREFIX = '#$'

MAIL_TYPE_MAPPING = {'ALL': 'abes', 'BEGIN': 'b', 'END': 'e', 'FAIL': 'as', 'NONE': 'n'}

mapping from MAVIS mail type options to SGE mail options

Type:dict

NAME = 'SGE'

the type of scheduler

Type:SCHEDULER

STATE_MAPPING = {'E': 'ERROR', 'R': 'RUNNING', 'T': 'ERROR', 'd': 'CANCELLED', 'h': 'PENDING', 'q': 'PENDING', 'r': 'RUNNING', 's': 'ERROR', 't': 'RUNNING', 'w': 'PENDING'}

mapping from SGE job states to their MAVIS JOB_STATUS equivalent

Type:dict

cancel(job, task_ident=None)

cancel a job or a specific task of an array job

Parameters:

• job (Job) – the job to cancel
• task_ident (int) – if specified, will cancel the given task instead of the whole array or job

classmethod convert_state(state)

format_dependencies(job)

returns a string representing the dependency argument

classmethod parse_qacct(content)

parses the information produced by qacct

Parameters:content (str) – the content returned from the qacct command

Raises

ValueError: when no job information is reported (this may happen due to a bad or too old job ID where information is no longer stored)

classmethod parse_qstat(content, job_id)

parses the qstat content into rows/dicts representing individual jobs

Parameters:content (str) – content returned from the qstat command

submit(job)

runs a subprocess sbatch command

Parameters:job (Job) – the job to be submitted

update_info(job)

runs a subprocess scontrol command to get job details and add them to the current job

Parameters:job (Job) – the job information is being gathered for

Raises

ValueError: if the job information could not be retrieved

class mavis.schedule.scheduler.SlurmScheduler(concurrency_limit=None, remote_head_ssh='')

Bases: mavis.schedule.scheduler.Scheduler

Class for formatting commands to match a SLURM scheduler system SLURM docs can be found here https://slurm.schedmd.com

Parameters:concurrency_limit (int) – the maximum allowed concurrent processes. Defaults to one less than the total number available

ENV_ARRAY_IDENT = 'SLURM_ARRAY_JOB_ID'

ENV_JOB_IDENT = 'SLURM_JOB_ID'

ENV_TASK_IDENT = 'SLURM_ARRAY_TASK_ID'

NAME = 'SLURM'

the type of scheduler

Type:SCHEDULER

cancel(job, task_ident=None)

cancel a job

Parameters:

• job (Job) – the job to be cancelled
• task_ident (int) – the task id to be cancelled (instead of the entire array)

format_dependencies(job)

returns a string representing the dependency argument

Parameters:job (Job) – the job the argument is being built for

classmethod parse_sacct(content)

parses content returned from the sacct command

Parameters:content (str) – the content returned from the sacct command

classmethod parse_scontrol_show(content)

parse the content from the command: scontrol show job <JOBID>

Parameters:content (str) – the content to be parsed

submit(job)

runs a subprocess sbatch command

Parameters:job (Job) – the job to be submitted

update_info(job)

Pull job information about status etc from the scheduler. Updates the input job

Parameters:job (Job) – the job to be updated

class mavis.schedule.scheduler.TorqueScheduler(concurrency_limit=None, remote_head_ssh='')

Bases: mavis.schedule.scheduler.SgeScheduler

Class for managing interactions with the Torque scheduler

Parameters:concurrency_limit (int) – the maximum allowed concurrent processes. Defaults to one less than the total number available

ENV_ARRAY_IDENT = 'PBS_JOBID'

ENV_JOB_IDENT = 'PBS_JOBID'

expected pattern for environment variables which store the job id

Type:str

ENV_JOB_NAME = 'PBS_JOBNAME'

expected pattern for environment variables which store the job name

Type:str

ENV_TASK_IDENT = 'PBS_ARRAYID'

MAIL_TYPE_MAPPING = {'ALL': 'abef', 'BEGIN': 'b', 'END': 'e', 'FAIL': 'fa', 'NONE': 'p'}

mapping from MAVIS mail type options to Torque mail options

Type:dict

NAME = 'TORQUE'

the type of scheduler

Type:SCHEDULER

STATE_MAPPING = {'C': 'COMPLETED', 'E': 'RUNNING', 'H': 'PENDING', 'Q': 'PENDING', 'R': 'RUNNING', 'S': 'ERROR', 'T': 'RUNNING', 'W': 'PENDING'}

mapping from Torque job states to their MAVIS JOB_STATUS equivalent

Type:dict

TAB_SIZE = 8

cancel(job, task_ident=None)

cancel a job

Parameters:

• job (Job) – the job to be cancelled
• task_ident (int) – if specified then a single task will be cancelled instead of the whole job or array

format_dependencies(job)

returns a string representing the dependency argument

classmethod parse_qstat(content)

parses the qstat content into rows/dicts representing individual jobs

Parameters:content (str) – content returned from the qstat command

submit(job)

runs a subprocess qsub command

Parameters:job (Job) – the job to be submitted

update_info(job)

runs a subprocess scontrol command to get job details and add them to the current job

Parameters:job (Job) – the job information is being gathered for

Raises

ValueError: if the job information could not be retrieved

mavis.schedule.scheduler.consecutive_ranges(numbers)

Given a list of integers, return a list of ranges

Example

>>> consecutive_ranges([1, 2, 3, 4, 5, 9, 10, 14, 18])
[(1, 5), (9, 10), (14, 14), (18, 18)]

mavis.schedule.scheduler.time_format(total_seconds)

Converts a total seconds to a str format “H:M:S”