Using Penn State's Roar Collab

Introduction

It is not often a simple, straigtforward task to identify the structural parameters of an economic model. First, the complexity of the model, makes the equilibrium conditions take longer to compute because sometimes they don’t have a closed form solutions. Second, the shape of the objective function which determines the model’s structural parameters also affects the computing time. For intense calculations, we may need to delegate the optimization routine to a high-performance computing environment instead of burdening our local machines, which could impede other activites like running lighter softwares, browsing, etc.

For advanced projects, including 2nd-year projects in ECON 512 (Empirical Methods in Economics), or RA projects, you might need to use a high-performance computing environment. This guide provides a step-by-step guide on how to use Penn State’s high-performance computing environment, Roar Collab, but the instructions can be externally valid to other systems using Slurm or MOAB/Torque job schedulers.

Eligibility to use the Supercomputers

To use the Roar Collab environment, you’ll need a faculty sponsor. If you’re in your 2nd year and don’t have an advisor yet, consider speaking with an ECON 512 instructor. Alternatively, discuss your motivation and need for high-performance computing with your advisor. You can request for an account here.

Preparing Files Locally

If you plan to use the supercomputer for running a script (e.g. MATLAB), ensure that you first test the script on your local machine. Also make sure that you have all the required files in the relevant directory. (The directory will be different in the supercomputer environment, so you’ll need to change the directory in the script as well.)

An example of the preamble of a MATLAB script is given below to set up the working directories.

  if strcmp(getenv('USER'), 'esma_pc') == 1
    cd('/Users/esma_pc/Projects/sample_proj/code');
  elseif strcmp(getenv('USER'), 'eqo5141') == 1
    cd('~/work/sample_proj/code');
  end

Connecting to the Linux Server

There are two ways to connect to the supercomputer and given your preference on using terminal you may choose among two.

1. Via the Web Portal

   Access the dashboard by logging in through this link. Once logged in, you’ll be able to navigate to files or applications with GUI.

2. Via Terminal

   Launch a terminal window on your local machine and execute the following command.

   ssh <your-username>@submit.hpc.psu.edu

   After entering your password, you’ll have access to the Roar Collab!

Uploading Your Files

To transfer your files on your local machine, you have several options. Using shared environments like Box, Dropbox, using git, or directly uploading are some of those. I’ll show the most direct (upload) approach. Suppose you’ve three types of files: a MATLAB script, a data input file, and a job submission file (more on this later).

1. Access the web portal.
2. Navigate to Files -> storage/work/<your username>.
3. Locate and click on the “Upload” button, which you’ll find at the top-right corner of the page.
4. Upload all three files on your work/ directory.

Testing the Script on an Interactive Session

After uploading your code scripts, it’s important to test it interactively before running it in the batch mode to avoid wasting significant time. (I’ve personally lost hours and even days due to this and it is the worst situation you can find yourself in, especially when you’re RAing.) If your code normally takes hours to run, consider modifying it for a quick test, such as reducing the loop iterations, if possible.

1. Access the web portal.
2. From the dashboard, select “My Interactive Sessions.”
3. In the left panel, select MATLAB and specify the resources you expect to use.
4. Click the “Launch” button. You’ll receive email notification once it is active.
5. Open the MATLAB session and run your script to verify it’s working in your new work environment.

Creating a Job Submission File

A job submission file is a text document that contains the information about the resources (nodes, time etc.) you plan to use and the script (s) you plan to run. This file has a .sbatch extension. Below is a template for creating your own.

#!/bin/bash

#SBATCH --nodes=8
#SBATCH --time=48:00:00
#SBATCH --mem=12gb

# Get started
echo "Job started on $(hostname) at $(date)"

# Load in matlab
module purge
module load matlab

cd <navigation to your working directory>
 
matlab -batch "your_matlab_file_name" >> logs/log.sample_matlab 

# Finish up
echo "Job ended at $(date)"

For more details on resource allocation, visit this link.

Submitting the Job

Once you’ve prepared all inputs for the job, you can proceed with its submission. You have the options to do this either from a terminal on your local machine or through the portal. We’ll use the portal.

1. From the dashboard, choose Clusters -> _RC Shell Access. A terminal window will appear.

2. Use the terminal commands (cd) to navigate to the directory containing your scripts and job submission file.

3. Execute the following command to submit your job.

   sbatch <job_submission_file_name>.sbatch.

Job Status

You’ve several options to check job status:

1. Via Terminal: Execute the following command:

   squeue -u <your_username>

2. Via dashboard: Navigate to Jobs -> Active Jobs on the dashboard.

Other useful job management commands can be found here.

Do’s and Don’ts

Do’s

1. Optimize Code: Ensure your code is as efficient as possible before submitting job. Every minute you invest in optimization will save you and others valuable computing time.
2. Terminal Commands: Familiarize yourself with terminal commands and consider editing your files directly in terminal using vim or nano. Terminal is faster than interactive desktop or apps with GUI.
3. Check Error Logs: If you face errors, inspect the *.e.* files in the submissoin folder and any log files specified in the .sbatch file.
4. Load Required Packages: Ensure all packages needed for your code are loaded. If you’re using non-default packages like knitro or cplex, include module load <package> in your .sbatch file.

Don’ts

1. Over-allocate resources: Never request more cores than necessary. If you’re not using parallel computing you don’t need more than one core.

Other Tips

1. If you need more than 48 hours of running time, consider talking to the Professor in charge to be considered for econ department queue. You’ll be able to use more resources (more running time and cores). Be very careful with using the resources (it is a paid allocation!) and be considerate of others.
2. Your jobs might sometimes be “suspended”. The reason is that queueing system places jobs on nodes that are used by users with paid allocations. Your job will be suspended until the resources are freed. Better to resubmit your job.