Skip to contents

SLURM Job Submission

slurm_job_submission.Rmd
library(abba)

In support for SLURM, abba has function to schedule and manage R programs on connected SLURM cluster. At a high level, these functions break down into three different categories:

  • Submit a job
  • Get job status
  • Wait for job completion
  • Retrieve the log of the job

Functions specifically for interfacing with SLURM follow the naming convention abba_slurm_*.

Submit a Job

Job submission is handled by the function abba_slurm_submit_job().

abba_slurm_submit_job(
  "/home/mike.stackhouse/repos/abba/test_programs/test_program.R"
  )
# [1] "38"

Note that this returns a vector with job id and name being path of the program. The Job ID is the identifier used downstream to interact with SLURM.

By default, program log is physically saved in the program folder. The log destination is customizable via log_path keyword parameter.

Get Job Status

Once the job is running in SLURM, abba can poll its status. This can be done using the function abba_slurm_get_job_status()

abba_slurm_get_job_status("57")
# 57 
# "COMPLETED"

The job statuses can be “RUNNING”, “COMPLETED”, FAILED etc. For a complete list of SLURM job statuses, please refer to documentation.

Get the Log Content

In this context, the “log” refers to the stdout/stderr of the program itself. This returns into a list object with the Job ID and the individual lines written out by the program. The content will contain all of the console output from the program itself.

abba_slurm_get_job_log("38")
# [[1]]
# [1] "[1] 4"                   "Warning message:"        "This is a test warning " "La-di-da"

Submit a batch of jobs to SLURM cluster

Here is an example of using SLURM submission functions in abba_submit_batch. Additional arguments like log_path will be passed on to submit_func

abba_submit_batch(list("/home/mike.stackhouse/repos/abba/test_programs/test_program.R", "/home/mike.stackhouse/repos/abba/test_programs/test_program.R"),
                  submit_func=abba_slurm_submit_job,
                  wait_func=abba_slurm_watch_job,
                  succeed_func=abba_slurm_get_job_succeeded,
                  log_path="/home/mike.stackhouse/test_logs")
# Submitting program:
#   /home/mike.stackhouse/repos/abba/test_programs/test_program.R
# Submitting program:
#   /home/mike.stackhouse/repos/abba/test_programs/test_program.R
# /home/mike.stackhouse/repos/abba/test_programs/test_program.R /home/mike.stackhouse/repos/abba/test_programs/test_program.R
                                                          "48"                                                           "49"