Python Simple Job Tutorial

Introduction

This tutorial is intended to familiarize you with the process of running and scheduling jobs on the SCRC Slurm cluster. You will create a slurm batch job to run a basic Python script that outputs the text “Hello World!”.

Make sure you’re logged in to a Linux terminal with SSH, see GETTING STARTED.

The Python Code

Using a text editing program such as nano or vim, create a python script ‘hello-world.py’.

'''

hello-world.py

Say Hi to the World.

'''

import time

# --------------------------------

# main

# --------------------------------

def main():

  print ("Hello World!")

  time.sleep(10)

if __name__ == '__main__':

  main()

The Slurm Script

#!/bin/bash

#

# [ hello-world.s ]

#

#

#SBATCH --job-name=hello-world             # job name

#SBATCH --mem=1g                           # request 1g memory

#SBATCH --mail-user=<netID>@stern.nyu.edu  # where to send email notifications

#SBATCH --mail-type=BEGIN,END,FAIL         # send email on job start, end, and fail

#SBATCH --output=hello-world-%j.out        # output file name; where to write terminal output; %j=jobID

#SBATCH --partition=test                   # use default partition "test"

#SBATCH --time=0-00:10:00                  # wallclock runtime: 10min

module purge                               # purge any loaded environment modules

module load python/3.9.7                   # load python

python hello-world.py                      # invoke your python program

Lines beginning with #SBATCH are Slurm directives that specify job options, resource requests, and scheduling parameters for a job submitted to a Slurm cluster.

The module command is used to manage environment modules, which allow users to load and unload software and environment settings on systems such as HPC clusters. The purge subcommand clears all loaded modules, while the load subcommand loads a specific module.

The python hello-world.py command executes the Python script using the loaded Python interpreter. The script is run with the version of Python that was loaded via the environment module.

Submitting the Job

To submit the job type:

sbatch hello-world.s

The job is now queued and as soon as the requested resources are available, Slurm schedules the job onto one of the Slurm compute nodes in the specified partition. If the requested resources cannot be met, the job waits in the queue. 

To check the status of your submitted jobs:

squeue -u $USER

To check all jobs in the queue and running on the cluster:

squeue