fsl.utils.fslsub
¶
This module submits jobs to a computing cluster using FSL’s fsl_sub command line tool. It is assumed that the computing cluster is managed by SGE.
Example usage, building a short pipeline:
from fsl.utils.fslsub import submit, wait
# submits bet to veryshort queue unless <mask_filename> already exists
bet_job = submit('bet <input_filename> -m',
queue='veryshort.q',
output='<mask_filename>')
# submits another job
other_job = submit('some other pre-processing step', queue='short.q')
# submits cuda job, that should only start after both preparatory jobs are
# finished. This will work if bet_job and other_job are single job-ids
# (i.e., strings) or a sequence of multiple job-ids
cuda_job = submit('expensive job',
wait_for=(bet_job, other_job),
queue='cuda.q')
# waits for the cuda job to finish
wait(cuda_job)
submit |
Submits a given command to the cluster |
info |
Gets information on a given job id |
output |
Returns the output of the given job. |
wait |
Wait for one or more jobs to finish |
func_to_cmd |
Defines the command needed to run the function from the command line |
-
fsl.utils.fslsub.
submit
(*command, minutes=None, queue=None, architecture=None, priority=None, email=None, wait_for=None, job_name=None, ram=None, logdir=None, mail_options=None, output=None, flags=False, multi_threaded=None, verbose=False)[source]¶ Submits a given command to the cluster
You can pass the command and arguments as a single string, or as a regular or unpacked sequence.
Parameters: - command – string or regular/unpacked sequence of strings with the job command
- minutes – Estimated job length in minutes, used to auto-set queue name
- queue – Explicitly sets the queue name
- architecture – e.g., darwin or lx24-amd64
- priority – Lower priority [0:-1024] default = 0
- email – Who to email after job completion
- wait_for – Place a hold on this task until the job-ids in this string or tuple are complete
- job_name – Specify job name as it will appear on the queue
- ram – Max total RAM to use for job (integer in MB)
- logdir – where to output logfiles
- mail_options – Change the SGE mail options, see qsub for details
- output – If <output> image or file already exists, do nothing and exit
- flags – If True, use flags embedded in scripts to set SGE queuing options
- multi_threaded –
Submit a multi-threaded task - Set to a tuple containing two elements:
- <pename>: a PE configures for the requested queues
- <threads>: number of threads to run
- verbose – If True, use verbose mode
Returns: string of submitted job id
-
fsl.utils.fslsub.
info
(job_id)[source]¶ Gets information on a given job id
Uses qstat -j <job_id>
Parameters: job_id – string with job id Returns: dictionary with information on the submitted job (empty if job does not exist)
-
fsl.utils.fslsub.
output
(job_id, logdir='.', command=None, name=None)[source]¶ Returns the output of the given job.
Parameters: - job_id – String containing job ID.
- logdir – Directory containing the log - defaults to the current directory.
- command – Command that was run. Not currently used.
- name – Job name if it was specified. Not currently used.
Returns: A tuple containing the standard output and standard error.
-
fsl.utils.fslsub.
wait
(job_ids)[source]¶ Wait for one or more jobs to finish
Parameters: job_ids – string or tuple of strings with jobs that should finish before continuing
-
fsl.utils.fslsub.
_flatten_job_ids
(job_ids)[source]¶ Returns a potentially nested sequence of job ids as a single comma-separated string
Parameters: job_ids – possibly nested sequence of job ids. The job ids themselves should be strings. Returns: comma-separated string of job ids
-
fsl.utils.fslsub.
func_to_cmd
(func, args, kwargs, tmp_dir=None, clean=False)[source]¶ Defines the command needed to run the function from the command line
WARNING: if submitting a function defined in the __main__ script, the script will be run again to retrieve this function. Make sure there is a “if __name__ == ‘__main__’” guard to prevent the full script from being rerun.
Parameters: - func – function to be run
- args – positional arguments
- kwargs – keyword arguments
- tmp_dir – directory where to store the temporary file
- clean – if True removes the submitted script after running it
Returns: string which will run the function