Python Simple Job Tutorial

Introduction

This tutorial is intended to familiarize you with the process of running and scheduling jobs on the SCRC Slurm cluster. You will use the Simple Python Job to run a basic Python script that outputs the text “Hello World!”.

Make sure you’re logged in to a Linux terminal with SSH. Instructions can be found at Accessing the SCRC Systems.

Writing Python code

Using a text editing program such as nano, vim, code, or emacs, create a python script ‘hello-world.py’.

'''
hello-world.py

Print string "Hello World!" to output

'''

# --------------------------------
# main
# --------------------------------  
def main():
  
  print("Hello World!")

if __name__ == '__main__':
  main()

Creating the Slurm Script

To run a Python script use a Slurm sbatch file to allocate the necessary compute resources. It contains directives for Slurm to interpret as well as programs for it to execute. Here is an example sbatch file, ‘hello-world.sbatch’, that submits the python job to the Slurm cluster. 

#!/bin/bash
#
# [hello-world.sbatch]
#
#SBATCH --job-name=hello              # Job name
#SBATCH --output=hello.out            # Output file name 
#SBATCH --export=ALL                  # Export all environment variables
#SBATCH --time=00:01:00               # Set max runtime of job = 1 minute
#SBATCH --mem=4G                      # Request 4 gigabytes of memory
#SBATCH --mail-type=BEGIN,END,FAIL    # Send email notifications
#SBATCH --mail-user=you@stern.nyu.edu # email TO
#SBATCH --partition=test              # Specify the partition to submit the job to


module purge                          # Start with a clean environment
module load python/3.9.7              # Load python module
python hello-world.py                 # Run the script using the loaded Python module

Lines beginning with #SBATCH are Slurm directives that specify job options, resource requests, and scheduling parameters for a job submitted to a Slurm cluster.

The module command is used to manage environment modules, which allow users to easily load and unload software and environment settings on systems such as HPC clusters. The purge subcommand clears all loaded modules, while the load subcommand loads a specific module.

The python hello-world.py command executes the Python script named “hello-world.py” using the Python interpreter. The script is run with the version of Python that was loaded via the environment module.

Executing the Slurm Script

Submit the job with the command:

sbatch hello-world.sbatch

The job is now queued and as soon as the requested resources are available, Slurm schedules the job onto one of the Slurm compute nodes. If the requested resources cannot be met, the job waits in the queue. To check the status of your submitted jobs:

squeue -u $USER

To check all jobs in the queue and running on the cluster:

squeue

Related Articles