BSUB Job Launcher

The BSUB class is a tool to write and launch job scripts on DOE HPC systems.

BSUB Class

class pbs4py.bsub.BSUB(project, ngpus_per_node=6, queue_node_limit=1000000, time=72, profile_filename='~/.bashrc', requested_number_of_nodes=1)

A Class for creating and running jobs using the Department of Energy batch system.

Parameters

  • project (str) – The project which to charge for submitted jobs

  • ngpu_per_node – The number of GPUs per compute node

  • time (int) – The requested wall time for the job(s) in hours

  • profile_filename (str) – The file setting the environment to source inside the PBS job

project: str

The project which to charge for submitted jobs

mail_when_complete: bool

Mail a job report when complete

property profile_filename

The file to source at the start of the pbs script to set the environment. Typical names include ‘~/.profile’, ‘~/.bashrc’, and ‘~/.cshrc’. If you do not wish to source a file, set to ‘’.

Type

str

mpiexec: str

mpiexec, mpirun, mpiexec_mpt, etc.

Type

The mpi execution command name

create_mpi_command(command, output_root_name, openmp_threads=1, ranks_per_node=None)

Wrap a command with mpiexec and route its standard and error output to a file

Parameters

  • command (str) – The command thats needs to run in parallel

  • output_root_name (str) – The root name of the output file, {output_root_name}.out.

  • openmp_threads (int) – The number of openmp threads per mpi process.

  • ranks_per_node (int) – The number of MPI ranks per compute node.

Returns

full_command – The full command string.

Return type

str

launch(job_name, job_body, blocking=True, dependency=None)

Create a job script and launch the job

Parameters

  • job_name (str) – The name of the job.

  • job_body (List[str]) – List of commands to run in the body of the job.

  • blocking (bool) – If true, this function will wait for the job to complete before returning. If false, this function will launch the job but not wait for it to finish.

  • dependency (str) – Jobs that this one depends one. For PBS, these are colon separated in the string

Returns

command_output – The stdout of the launch command. If the job is successfully launch, this will be the job id.

Return type

str

property mpiprocs_per_node

The number of requested mpiprocs per node. If not set, the launcher will default to the number of cpus per node. #PBS -l select=1:ncpus=40:mpiprocs={mpiprocs_per_node}.

Type

int

property requested_number_of_nodes

The number of nodes to request. That is, the ‘select’ number in the #PBS -l select={requested_number_of_nodes}:ncpus=40:mpiprocs=40.

Type

int

write_job_file(job_filename, job_name, job_body, dependency=None)

Create a launch script file in the current directory for the commands defined in job_body.

Parameters

  • job_filename (str) – name of file to write to

  • job_name (str) – The name of the job.

  • job_body (List[str]) – List of commands to run in the body of the job.

  • dependency (str) – Jobs that this one depends one. For PBS, these are colon separated in the string

hashbang: str

The hashbang line which sets the shell for the PBS script. If unset, the default is #!/usr/bin/env {self.shell}.

shell

The shell flavor to use in the PBS job

queue_node_limit: int

The maximum number nodes allowed by the queue

time: int

The requested wall time for the pbs job(s) in hours

ncpus_per_node: int

The number of CPU cores per node.

ngpus_per_node: int

The number of GPUs per node.

ranks_per_node_flag: str

Command line option for mpiexec to specify the number of MPI ranks for host/node. Default is to set it based on the mpiexec version.

tee_output: bool

If true, redirection of the output of mpi commands changed to tee