Python Virtual Environment

A Python virtual environment is a personal isolated Python installation. The virtual environment consists of a directory that contains all the necessary executables for a Python project. You can customize this environment by installing additional Python packages.

General Steps: Creating or Configuring

  1. On a login node, create python virtual environment in your bigdata directory. First time only.
  2. Activate
  3. Install packages
  4. Deactivate

Running as Slurm batch job

In sbatch script, include module load and virtual environment activation, e.g.,

#!/bin/bash
#
# [hello-world.sbatch]
#
#SBATCH --job-name=hello              # Job name
#SBATCH --output=hello.out            # Output file name 
#SBATCH --export=ALL                  # Export all environment variables
#SBATCH --time=00:01:00               # Set max runtime of job = 1 minute
#SBATCH --mem=4G                      # Request 4 gigabytes of memory
#SBATCH --mail-type=BEGIN,END,FAIL    # Send email notifications
#SBATCH --mail-user=you@stern.nyu.edu # email TO
#SBATCH --partition=test              # Specify the partition to submit the job to


module purge                                      # Start with a clean environment
module load python/3.9.7                          # Load python module
source  ~/bigdata/05-virtenvs/py3.9/bin/activate  # activate virt env
python hello-world.py                             # Run python

Submit job:

sbatch hello-world.sbatch

How to Create a Python virtual environment

Note: Always create virtual environments in your bigdata directory. If you do not have a bigdata directory, email scrc-list@stern.nyu.edu to request one.

Load the Python module.

module load python/3.9.7

Create the virtual environment and specify the location. For example

virtualenv ~/bigdata/05-virtenvs/py3.9

You only need to create the virtual environment once but you can create multiple virtual environments with different versions and flavors of Python.

Activate

source ~/bigdata/05-virtenvs/py3.9/bin/activate

Notice the name of the active virtual environment is prepended to your prompt

Install Python modules with pip

See modules currently installed

pip list

Install a new module

pip install requests

Deactivate

deactivate
Related Articles