UsePBS

How do I use the PBS submission system?

Link to Torque documentation from ClusterResources

Initializing the PBS system

To enable the job submission system you need to first add PBS to the list of available modules.

module add torque

if the above command results in an error, then please add the following snippet to your .bashrc file.

# Setup the torque parallelization queue:
if [ $BASH ] && [[ $TERM == xterm* ]]
then
    source /etc/profile.d/modules.sh
    if [ `hostname` == "ba3" ]
    then
        module add torque
    fi
fi

Submitting jobs

There are multiple ways to submit jobs.

1. Easy way:

    /software/common/bin/ezsub -c "sleep 20" -n my_sleep_job
   

2. More flexible but complex way:

    echo "sleep 20" | qsub -V -N -m abe my_sleep_job
   

3. Even more flexible and complex way. First create a shell script (mysleepscript.sh):

   #!/bin/bash
   #PBS -V
   #PBS -N my_sleep_job
   #PBS -m abe
   sleep 20

   Then execute via qsub:

    qsub mysleepscript.sh
   

When you execute qsub processes above, it will send you emails indicating the submission, start and end of jobs. It will also generate files storing the stdout and stderr from each job. These will be stored in different locations depending on how you submitted the job. For (1) above it will be in $HOME/pbs and for 2 and 3 it will be in the job submission directory.

Performing interactive jobs

Log into your lab's login node via nx, vnc or ssh and then execute:

qsub -IVX -q interactive

The I says it's interactive, V says use my environment variables an X says export display.

Submitting MATLAB jobs to the queue

First, you need to know how to run a matlab job in the background, without worrying about the queue system. Because you don't have a screen on which to display things, you aren't going to want the desktop loaded, and you aren't going to want the splash screen loaded. You'll want your commands in an inputfile, and you'll want the matlab output to be saved in its own file. We'll accomplish both these things using Unix redirection commands.

    matlab -nodesktop -nosplash < mymatlabcommands.m > mymatlaboutput.txt

which is interpreted as: "Launch matlab without the desktop or splash screen. Run the commands in the mymatlabcommands.m file. Write the text output into mymatlaboutput.txt. Then quit."

As a simple example, we have a textfile, mymatlabcommands.m:

    pwd
    display('hello world!');

Now when we run the command above, we generate an output file that shows the current working directory and a greeting to the world.

If we want to run this job in the background on the local server, we can simply add an & to the end of the command:

    matlab -nodesktop -nosplash < mymatlabcommands.m > mymatlaboutput.txt &

With this framework, running this as a queue process becomes quite simple. Create a shell script much like the one above:

#!/bin/bash
#PBS -V
#PBS -N my_matlab_job
#PBS -m abe
cd ~/the_directory_where_I_want_matlab_to_start
matlab -nodesktop -nosplash < mymatlabcommands.m > mymatlaboutput.txt

Then execute via qsub:

    qsub mymatlabscript.sh

Caveats: any output that requires a graphical output (plots, figures, motion graphs, et cetera) will not be displayed, obviously.

Checking job/pbs status

1. pbsnodes: This command will give you the status of the queueing system.
2. qstat: This will give you the status of the running jobs. The -f and -a flags give more/different info.
3. ~tkp/bin/iqstat: This will open an interactive terminal that will provide an interactive view of the status of the queue and running jobs. (If you do not have access to an X display, you can use ~tkp/bin/iqstat.sh instead.)

Deleting jobs

1. Delete all your jobs:

    qdel all
   

2. Delete a specific job:

    qdel job_id (you can see your job ids by looking at qstat or iqstat)
   

   This is why it's important to name your job to reflect what it might be doing.

3. Delete all your jobs one at a time (this is helpful if you have submitted a large number of jobs, where "qdel all" might cause problems for the submission server)

    ~tkp/bin/qdel-all-my-jobs.sh
   

4. You can also delete all of your jobs whose name includes a certain string:

    ~tkp/bin/qdel-all-my-jobs.sh estimate_model