Python Virtual Environment

A Python virtual environment is a personal isolated Python installation. The virtual environment consists of a directory that contains all the necessary executables for a Python project. You can customize this environment by installing additional Python packages.

These instructions are for creating the Python virtual environment, typically, done once. Once created, activate from within a slurm batch script or inside an interactive slurm job. 

How to Create a Python Virtual Environment

Always create virtual environments in your ~/bigdata directory. If you do not have a ~/bigdata directory, email scrc-list@stern.nyu.edu to request one.

Example

Create Virtual Environment

Load the python environment from which to create the virtual environment. Create a directory for the virtual environment. Create the virtual environment.

module load python/3.9.7

mkdir ~/bigdata/05-virtenvs

python -m venv ~/bigdata/05-virtenvs/py3.9

Activate

To activate:

source ~/bigdata/05-virtenvs/py3.9/bin/activate

(py3.9)

[user123@login-node-1 ~] $

Notice the name of the active virtual environment is prepended to your prompt

To List and Install Python Packages

pip list

pip install duckdb

To Run a Python Program in This Virtural Environment

python <my-prog.py>

Deactivate

deactivate

Run you python code in a batch or interactive Slurm job on the SCRC slurm cluster or from your research project virtual machine.
Do not run analyses directly on a login node.

E.g., For a simple python batch job, include module load and virtual environment activation in the slurm submission script:

#!/bin/bash

#

# [hello-world.s]

#

#SBATCH --job-name=hello-world             # job name

#SBATCH --mem=1g                           # request 1g memory

#SBATCH --mail-user=<netID>@stern.nyu.edu  # where to send email notifications

#SBATCH --mail-type=BEGIN,END,FAIL         # send email on job start, end, and fail

#SBATCH --output=hello-world-%j.out        # output file name; where to write terminal output; %j=jobID

#SBATCH --partition=test                   # use default partition "test"

#SBATCH --time=0-00:10:00                  # wallclock runtime: 10min

module purge                               # purge any loaded environment modules

module load python/3.9.7                   # load python

source ~/bigdata/05-virtenvs/py3.9/bin/activate  # activate virt env

python hello-world.py                      # run your python program