GET STARTED

Getting An Account

Stern Users

For those in the Stern School of Business, you can use your existing activated Stern account, i.e., Stern NetID and password, to access SCRC services and resources.

Non-Stern NYU Users

Those already at NYU but not in Stern (and not enrolled in a Stern course) must have a full-time Stern employee make a request on their behalf by contacting the Stern IT Service Desk at servicedesk@stern.nyu.edu or 212-998-0180 and provide the following information:

Full Name
NYU NetID
NYU UniversityID (N#)
Date of birth
Sponsor’s name (Stern faculty or employee)
Sponsoring department
Expiration date (max 1 year; extensions may be requested yearly)

Once you have a valid Stern account, be sure to Activate your Stern account.

Non-NYU Affiliated Users

Non-NYU affiliated users must first obtain an NYU account by having an authorized sponsor request an NYU NetID on their behalf by filling out the IT Onboarding Form. It is accessed by logging into the NYU VPN, then logging into identity.it.nyu.edu , clicking the hamburger menu, selecting Request Affiliate, and then Request Affiliate User. For additional help contact NYU IT support at AskIT@nyu.edu or 212-998-3333.

Accessing the SCRC Systems

Log in to a Terminal Command Line Interface

The most common way to access the SCRC servers is to use a Command Line Interface (CLI) from a terminal application. A CLI is a text-based interface that allows users to interact with a host by entering text commands at the prompt of a terminal application. The SCRC hosts that you connect to are running Linux.

You will use your Stern NetID and password to authenticate to one of SCRC's login nodes:

rnd.scrc.nyu.edu

vleda.scrc.nyu.edu

If Off Campus, First Connect to the NYU VPN

If you are connecting from a remote location that is not on the NYU network (your home for example), you need to set up your computer to use the NYU VPN. Once you've created a VPN connection, you can proceed as if you were connected to the NYU network. If you're on the NYU network already, you can proceed directly with your connection.

Open the terminal application and use the ssh command to make a secure connection to one of SCRCs remote login hosts. E.g., if your Stern NetID is xy12:

ssh xy12@rnd.scrc.nyu.edu

ssh xy12@vleda.scrc.nyu.edu

Windows users have several options:

Command Line: Open the cmd program on an updated Windows 10 or later machine. Type the ssh command the same as shown above for Mac/Linux.
WSL2: If you run Windows 10 or later, you can install Windows Subsystem for Linux (WSL), and then install Ubuntu or other Linux distributions on your machine. You will get a fully functional Ubuntu distribution with a terminal application. Type the ssh command the same as shown above for Mac/Linux.
PuTTY: PuTTY is a free and open-source terminal emulator and network file transfer application that supports logging in with the ssh command.
- Note: If you are using WSL2, you may not be able to access the internet when Cisco AnyConnect VPN, installed from the .exe file, is activated. A potential solution is to uninstall Cisco AnyConnect and install AnyConnect using Microsoft Store and then set up a new VPN connection using the settings described onthe IT webpage .

Getting and using PuTTY

PuTTY is free and can be downloaded from PuTTY Download.

Download and run the latest installer file from the website.
Follow the on-screen install instructions.
Once installed, start PuTTY, then type rnd.stern.nyu.edu or vleda.stern.nyu.edu in the Host Name field and click Open.
Note: The first time you log in a dialog may appear which starts with The server's host key is not cached. Click Yes in this dialog.
Type your Stern NetID and password at the login as and password prompts, respectively.

You should now be logged in and see a command line prompt from which you can enter Linux commands or run programs.

Transferring Files from Local to Remote Linux Server

This guide explains how to transfer files between your local computer and a remote Linux server.

Using FileZilla (Windows, Linux, and Mac):

Download, install, and start FileZilla.
If off-campus, first log in to the NYU VPN.
Enter the following connection details:
- Host: rnd.scrc.nyu.edu
- Username: <your Stern ID>
- Password: <your password>
- Port: 22
Click "Quickconnect."
On the left side of FileZilla, you should see your local computer. On the right side, you should see your Linux home directory.
Drag and drop files to transfer between your local computer and the Linux server.

Linux Tutorial

For an in-depth tutorial, you can navigate to this Linux guide.

This tutorial demonstrates the basic Linux commands a user would need to use and run programs on the SCRC Linux computers. The Linux machines use a command line interface called a Shell. The Shell is the interface where you enter Linux commands. There are many types of shells. By default, SCRC Linux machines use the BASH shell.

The first thing you see after logging in is the shell's command prompt. For example, Christine Thomass' prompt might look something like this:

[ct27@rnd ~ ]$

Your Home Directory

Once logged in, you are deposited into your home directory. Every user has a home directory. Your home directory is where you will store most of your files.

Show the present working directory

To see the complete name and path of the current working directory, use the command

pwd

Example output:

/homedir/employees/c/ct27

This prints the full directory path. Here, Christine's home directory name is ct27 which is contained within a directory called c which is inside the directory employees inside the directory homedir. The Linux directory structure is a hierarchical tree structure.

Show contents of a directory

To see a list of the files and directories in the current directory type ls -l (list directory contents using long format), e.g.,

ls -l

Example output:

total 180

drwxr-xr-x 3 ct27 nobody 4096 Apr 5 2023 mywork

drwxr-xr-x 3 ct27 nobody 4096 Apr 3 2023 archived-work

-rw-r--r-- 1 ct27 nobody 0 Mar 18 2023 prog.sas7bdat

-rw-r--r-- 1 ct27 nobody 1421 Mar 18 2023 prog.sas7bdat.log

ls is the command and -l is, in this example, the option for the command. The -l option formats the results into long format. Long format lists more than just the names of the files which is what ls alone would show.

In this example, the output shows two directories mywork and archived-work (notice the d at the beginning of the line), and two files prog.sas7bdat and prog.sas7bdat.log. Some other information we see, e.g., the size of the log file is 1421 bytes, its owner is ct27, its last modification date is Apr 5 2023.

Auto Completion in the Shell

Press the Tab key after enough of the word you are trying to complete has been typed in. If when hitting tab the word is not completed there are probably multiple possibilities for the completion. Press tab again and it will list the possibilities.

Create a sub-directory

Say you would like to work on a project. It is probably convenient to create a sub-directory to hold the projects files, i.e., program, data and results. To create a directory use the mkdir (make directory) command, e.g.,

mkdir costProject

This command creates a new directory called costProject inside the current directory. Verify this by typing ls -l.

Note: Linux is case sensitive, so for example costproject is not the same as costProject.

Change to a subdirectory

To change into the costProject directory, i.e., make it the current directory, use the cd (change directory) command, e.g.,

cd costProject

Verify that we are in the costProject directory using pwd.

pwd

Example output:

/homedir/employees/c/ct27/costProject

Create or edit a file

nano is a simple text editor used to create or edit a data or program file, etc. For example, to edit a file called costdata.dat type,

nano costdata.dat

Along the bottom of the nano editor are the editor's commands. The character ^ is the ctrl key. To save the file you type ctrl-o and then to exit nano type ctrl-x.

List the contents of a file

The more command displays the contents of a file, one screen-full at a time. Press the space bar to move forward screen by screen until the end of the file is reached or press the letter q, for quit., E.g.,

more costdata.dat

If the file contents are larger than can be shown in one page, the more command shows you the first page and halts. To see the next page press the space bar and to quit type q.

Breaking out

If you get stuck or hung after typing a command, etc., then use ctrl-c to break out or cancel your current command. ctrl-c returns you to the command prompt.

Commands

Most Linux commands are short, typically an abbreviation or mnemonic for what the command does. For example, the command to delete a file is rm which is short for remove.

Recall: Linux is case-sensitive, i.e., it distinguishes between lowercase and uppercase (recognizing commands in lowercase only).

Most commands follow a common syntax:

command -options arguments

Here are some examples of basic Linux commands and command syntax.

To list all of the files in your current directory, type
ls
To get a long listing of the files in your current working directory including information on size, date of modification and permissions, type
ls -l
To make a copy of a file and give it a new name, type
cp costdata.dat costdata.dat.bak
To remove (delete) a file, type
rm costdata.dat
Note: There is no undo in Linux therefore once a file has been deleted there is no easy way to recover it.
To rename (move) a file, type
mv costdata.dat.bak newcostdata.dat
To create a sub-directory, type
mkdir study1
To change (move down) into the new directory, type
cd study1
To change (move up) to the previous directory, type
cd ..
To remove a directory, type
rmdir study1
To remove ALL files in the current directory whose name begins with costdata, type
rm costdata *
The asterisk (*) is a wildcard character that matches one or more characters. Be VERY careful with removing files and using the * wildcard character. You might end up permanently deleting more than you intended. Always do a list first, e.g., ls costdata* to show the files that would be deleted with the rm costdata* command.

Redirecting Input and Output

Commands usually display their results to the screen. Also, commands normally operate on data as you type it in from the keyboard. A right angle-bracket > (called an "into") on the command line indicates that the next word is the name of a file or device in which to place, or redirect the output of a command, e.g.,

ls > list

will place the output of the ls command in a file named list. If a file named list existed before you entered this command, any previous contents will be over written.

You can append to the end of a file using a double right angle-bracket >> (called an "onto"). For example, if the next command entered was:

date >> list

The output of the date command would be added to the bottom of the file called list.

Pipes

The output of one command can be fed as input into to another command. The symbol for the input/output (I/O) connection is a vertical bar | called a pipe.

For example, a directory containing many files will scroll off the screen if just ls is used. If a pipe is used to direct the output of ls to be the input of more, then the directory listing will be shown one screen-full at a time and will not scroll off the screen.

ls | more

Online Help

To get on-line help in Linux, use the man command. It provides access to a comprehensive online Linux manual. Type man followed by a command or topic on which you want help.

Online Manual

To make the online manual more helpful, an index is provided. The index is accessed with the -k option of the man command.

Example:

man -k directory

This will display a one-line synopsis of all manual pages having to do with directories.

The Linux File System

Linux uses a hierarchical file structure, which is made up of files, directories, and sub-directories.

A file can hold text, data, or a program.
Directories contain files and sub-directories.
A sub-directory is a directory that has been created within another directory.

You can also use the --help option to get a description of the commands usage. Simply type your command whose usage you to know in the terminal with --help after a space and press enter. And you'll get the complete usage of that command as shown below.

An example below:

cat --help

File Permissions

Since Linux is set up to let users share files, you have the option of allowing or denying access to others on the system. Permissions determine who may access your files and directories or what may be done with a file or a directory.

Use ls -l to see what permissions your files and directories have.

Example output:

-rw------- 1 ct27 devel Aug 20 10:15 logon

-rwx------ 1 ct27 devel Aug 19 15:23 a.out

-r-xr-xr-x 1 ct27 devel Aug 28 09:48 ls-list

drwx------ 1 ct27 resch Aug 27 15:45 sas-one/

drw------- 1 ct27 resch Aug 27 15:45 tex/

The characters d, r, w, x, and - at the far left indicate the permission of each file and directory. There are 10 positions.

Position 1: The directory indicator.
Positions 2,3,4: Apply to the owner (creator).
Positions 5,6,7: Apply to the group.
Positions 8, 9,10: Apply to all users.

The meaning of the letters are:

d (directory): If the first letter is a d, the file is a directory. If the first character is a hyphen (-), then it is a regular file.
r (readable): A file must be readable to be looked at or copied. A directory must be readable for you to list its contents.
w (writable): A file must be writable in order for you to modify it, remove it, or rename it. A directory must be writable in order for you to add or delete files in it.
x (executable): A file with executable permissions is one you can run, such as a program or a shell script. A directory must be executable in order for you to move into it (using the cd command), list its contents, or create or delete files there.
- (hyphen): Appears when the permission is switched off.

Examples

The file is read/write for owner, read only for others: -rw-r--r--
The directory is read/write/search for owner, read/search for group, searchable only for others: drwxr-x--x

Changing File and Directory Permissions

The file and directory permission levels in Linux can be changed to allow or deny access to other users. The chmod command is used to set the protection level.

You can add or remove protection levels by using either + (add) or - (remove) with chmod and a letter signifying a class of users:

u: for the file owner
g: for a system defined group
o: for users in neither u or g
a: for all users

and the desired access settings:

r: for read access
w: for write access
x: for execute access

Example:

To change the file called my.dat with the permissions -rw-r--r-- to read/write/execute for all users (owner, group, and other) type the command:

chmod a+rwx my.dat

The resultant permissions would be: -rwxrwxrwx

Now, change the permissions of my.dat so that only the owner can read, modify or delete it:

chmod u-x my.dat

chmod g-rwx my.dat

chmod o-rwx my.dat

The resulting permissions are: -rw-------

Command History

The most recent commands typed at the command line interface are saved and can be viewed or retrieved for re-use. Access the history by pressing the up arrow.

To search for a command in the history type ctrl-r, type the search text, then press enter to re-execute or ctrl-c to abort.

Basic Cursor Movement, Cut, Paste and Undo

ctrl-a move cursor to beginning of line

ctrl-e move cursor to end of line

ctrl-k cut everything after the cursor

ctrl-y paste the last thing cut

ctrl-_ undo

Command Summary

cat display contents of a file

cd dir moves you to directory called dir

pwd display the current working directory

chmod permissions sets permissions on a file

clear clear screen

cp file1 file2 copies a file

diff file1 file2 compare two files

exit exit Linux shell

file lists file type of given file

find name filename search for file called filename

grep searchstring filename search files for search string

head displays top lines of a file

history displays recently entered commands

ls lists the contents of a directory

ls -l like ls, but in long format

man cmd displays manual pages about cmd

mkdir dir creates a directory

Python Simple Job Tutorial

Introduction

This tutorial is intended to familiarize you with the process of running and scheduling jobs on the SCRC Slurm cluster. You will use the Simple Python Job to run a basic Python script that outputs the text Hello World!. Make sure you are logged in to a Linux terminal with SSH. Instructions can be found at Accessing the SCRC Systems.

Writing Python Code

Using a text editing program such as nano, vim, code, or emacs, create a python script hello-world.py.

'''

hello-world.py

Print string "Hello World!" to output

'''

# --------------------------------

# main

# --------------------------------

def main():

print("Hello World!")

if __name__ == '__main__':

main()

Creating the Slurm Script

To run a Python script use a Slurm sbatch file to allocate the necessary compute resources. It contains directives for Slurm to interpret as well as programs for it to execute. Here is an example sbatch file, hello-world.sbatch, that submits the python job to the Slurm cluster.

#!/bin/bash

# [hello-world.sbatch]

#SBATCH --job-name=hello # Job name

#SBATCH --output=hello.out # Output file name

#SBATCH --export=ALL # Export all environment variables

#SBATCH --time=00:01:00 # Set max runtime of job = 1 minute

#SBATCH --mem=4G # Request 4 gigabytes of memory

#SBATCH --mail-type=BEGIN,END,FAIL # Send email notifications

#SBATCH --mail-user=you@stern.nyu.edu # email TO

#SBATCH --partition=test # Specify the partition to submit the job to

module purge # Start with a clean environment

module load python/3.9.7 # Load python module

python hello-world.py # Run the script using the loaded Python module

Lines beginning with #SBATCH are Slurm directives that specify job options, resource requests, and scheduling parameters for a job submitted to a Slurm cluster.

The module command is used to manage environment modules, which allow users to easily load and unload software and environment settings on systems such as HPC clusters. The purge subcommand clears all loaded modules, while the load subcommand loads a specific module.

The python hello-world.py command executes the Python script named hello-world.py using the Python interpreter. The script is run with the version of Python that was loaded via the environment module.

Executing the Slurm Script

Submit the job with the command:

sbatch hello-world.sbatch

The job is now queued and as soon as the requested resources are available, Slurm schedules the job onto one of the Slurm compute nodes. If the requested resources cannot be met, the job waits in the queue.

To check the status of your submitted jobs:

squeue -u $USER

To check all jobs in the queue and running on the cluster:

squeue

Page updated

Report abuse