LoadLeveler

SP Parallel Programming Workshop

l o a d l e v e l e r

Table of Contents

What Is LoadLeveler?
LoadLeveler Overview
Basic LoadLeveler Tasks
Submitting Multiple Jobs
Using the Job Command File as the Executable
Submitting Parallel Jobs - General Notes
Submitting MPI Parallel Jobs
Submitting PVM Parallel Jobs
Submitting PVMe Parallel Jobs
LoadLeveler Internals
How LoadLeveler Schedules Parallel Jobs
LoadLeveler at the MHPCC
LoadLeveler Job Command File Keywords Reference
LoadLeveler Commands Reference
References and More Information
Exercise

What Is LoadLeveler?

LoadLeveler is a batch job scheduling application and program product of IBM.
Provides the facility for building, submitting and processing batch jobs within a network of machines
LoadLeveler scheduling matches job requirements with the best available machine resources
In a multi-user production environment, the use of LoadLeveler is intended to promote overall improved system performance, turnaround time and equitable resource distribution for all users
Flexible - permits each machine in the pool to be configured differently. For example: a machine may be configured to be used for batch work only during off-work hours.
Can schedule serial or parallel (MPI) jobs
LoadLeveler environment can include all of the following:
- Machines of different architectures:
  - IBM RS/6000
  - IBM 9076 (SP) Systems
  - Sun SPARCstations
  - Silicon Graphics IRIS
  - HP Apollo 9000 series workstations
- Submit only machines
- Batch only (dedicated) machines
- Interactive batch machines
- Can be configured to schedule jobs for NQS machines outside the LoadLeveler pool (see LoadLeveler Administration Guide for details).
Provides a graphical user interface called xloadl for job submission and monitoring

LoadLeveler Overview

The entire collection of machines available for LoadLeveler scheduling is called a "pool"
Every machine in the pool has one or more LoadLeveler daemons running on it.
There is one Central Manager machine for the LoadLeveler pool
- Principal function is to coordinate LoadLeveler related activities on all machines in the pool
- Maintains status information on all machines and jobs - makes decision on where jobs should be run
- If Central Manager machine goes down
  - Job information is not lost.
  - Jobs executing on other machines continue to run
  - Jobs waiting to run will start when Central Manager is restarted
  - May continue to submit jobs from other machines - will be dispatched when Central Manager is restarted
- A backup Central Manager can be designated - automatically assumes Central Manager responsibilities if it goes down.
- Normally, users do not need to even know about the Central Manager
Other machines in the pool may be used to perform one or more of the following tasks
- Submit jobs
- Execute jobs
- Schedule submitted jobs (in cooperation with Central Manager)
Every LoadLeveler job must be defined in a job command file
Only after defining a job command file, may a user submit the job for scheduling and execution. (Construction of a job command file is discussed in more detail next)

Basic LoadLeveler Tasks

Building a Job Command File

Every LoadLeveler job must be submitted through a job command file. LoadLeveler will not directly accept executable (a.out) files.
Resembles a shell script in appearance
The Job command file contains LoadLeveler statement lines with LoadLeveler keywords that describe the job to run. May also contain comment lines (not executed).
LoadLeveler keywords specify job information such as
- Executable name
- Class (queue)
- Resource requirements
- Input/output files
- Number of processors required
- Job type (serial, parallel, pvm3)
With the LoadLeveler GUI (xloadl), jobs can be submitted by using the "File" menu on any of the 3 xloadl windows.

A simple example job command file appears below


# The executable is myprog.  The stdin, stdout and stderr
# files are named accordingly.
#
#@ executable = myprog
#@ input      = myprog.in
#@ output     = myprog.out
#@ error      = myprog.err
#@ account_no = ABCDE-1234-567
#@ queue

The following general rules apply to job command files:
- Statements begin with a # @ followed by LoadLeveler keywords and statement components. Any number of blanks may separate the # @ and statement components.
- Keywords are case insensitive. They may be entered in lowercase, uppercase or mixed case.
- Comment lines begin with a #
- You may name your job command file whatever you like. One convention, which is used by the LoadLeveler GUI (xloadl), is to add a suffix ".cmd" to the filename. This is not required, however.

Basic LoadLeveler Tasks

Submitting a Job

The llsubmit command is used to submit job command files for scheduling and execution. For example, if your job command file was called, "myjob.cmd" you would submit it to LoadLeveler by using the command:
```
    llsubmit myjob.cmd
```

LoadLeveler will then respond to you with something like:


    submit: 1 job has been submitted, "fr2n02.32.0"

The name that LoadLeveler assigns to your job is unique and consists of three parts:
- machine name that participates in the scheduling of the job - typically the name of the machine where you issued the llsubmit command. In the example above, fr2n02 is the machine name.
- cluster id - identifier assigned by LoadLeveler to a group of jobs that are initiated from same job command file. In the example above, 32 is the cluster id.
- process id - unique for every job. Begins at zero and increments by one for each job that originates from the same command file. For example, if five jobs had been queued from the same command file, the process ids would be fr2n02.32.0 through fr2n02.32.4 .
See the llsubmit man page for details
With xloadl, jobs can be submitted by using the "File" menu on any of the 3 xloadl windows.

Basic LoadLeveler Tasks

Displaying Job Status

The llq command is used to obtain information about jobs in the LoadLeveler job queue.
The default is for llq to display information about all jobs in the queue
Other options include the ability to display information by userids, hostnames, queue host, cluster id and process id.
A detailed (long) list is displayed if the -l option is specified
See the llq man page for details
With xloadl, job status information can be displayed in the "Jobs" window, with selection and sort options provided by the "Select" and "Sort" menus

A sample short (default) list appears below

Id Owner Submitted ST PRI Size Class Running On ------------ ------- ----------- -- --- ---- --------- ----------- fr3n05.14.0 rjz 7/9 09:13 R 50 0.0 large fr7n09 fr7n05.15.0 bmr3 7/9 10:34 R 50 0.0 large fr7n15 fr5n09.13.4 ksmith 7/9 11:05 R 50 0.0 medium fr3n13 fr5n09.13.3 ksmith 7/9 11:05 R 50 0.0 medium fr7n03 fr5n09.13.2 ksmith 7/9 11:05 P 50 0.0 medium fr5n09.13.0 ksmith 7/9 11:05 I 50 0.0 medium fr5n09.13.1 ksmith 7/9 11:05 I 50 0.0 medium fr2n09.33.1 sokel 7/9 12:19 I 50 0.0 bigmem fr3n11.02.1 salay 7/9 12:23 I 50 0.0 bigmem fr3n01.02.1 caldwel 7/8 02:49 H 50 0.0 large 10 jobs in queue 4 waiting, 1 pending, 4 running, 1 held.

Basic LoadLeveler Tasks

Changing a Job's Priority

Use the llprio command to change the priority of one or more jobs in the LoadLeveler queue
To adjust the priority, supply a + (plus) or - (minus) followed by an increment value
Priority of a job ranges from 0 to 100, with higher numbers corresponding to greater priority
Only the owner of a job or the administrator can change the priority of a job
llprio does not effect a job that is running, even if its priority is lower than other jobs
For example, to increase the priority of a job by 10:
```
    llprio +10 mymachine.23.0
```
To specify an absolute priority, use the -p option:
```
    llprio -p 75 mymachine.23.0
```
See the llprio man page for details
With xloadl, a job's priority may be changed by using the "Actions" menu in the "Jobs" window

Basic LoadLeveler Tasks

Holding a Job

Use the llhold command to place a temporary hold on a job in the queue
Does not affect running jobs - only jobs in the queue
Only the LoadLeveler system administrator can put other user jobs hold.
For example, to put a hold on job mymachine.23.0:
```
    llhold mymachine.23.0
```
See the llhold man page for details
With xloadl, jobs may be put on a hold by using the "Actions" menu in the "Jobs" window

Releasing a Held Job

Use the llhold command with the -r option to release a job that is on hold
Only the LoadLeveler system administrator can release other user jobs which are on hold.
For example, to release the hold on job mymachine.23.0:
```
    llhold -r mymachine.23.0
```
See the llhold man page for details
With xloadl, jobs may be released from a hold by using the "Actions" menu in the "Jobs" window

Basic LoadLeveler Tasks

Displaying a Machine's Status

Use the llstatus command to display information about machines in the LoadLeveler pool

By default, llstatus will display one line of information for each machine in the pool. An example appears below:

Name Schedd InQ Act Startd Run LdAvg Idle Arch OpSys cws1.mhpcc.edu Down 0 0 Down 0 0.00 22 R6000 AIX41 fr1n05.mhpcc.edu Avail 0 0 Busy 1 0.00 9999 R6000 AIX41 fr1n07.mhpcc.edu Avail 0 0 Idle 0 0.07 9999 R6000 AIX41 fr2n01.mhpcc.edu Avail 0 0 Busy 1 0.00 9999 R6000 AIX41 fr2n05.mhpcc.edu Avail 0 0 Busy 1 1.00 9999 R6000 AIX41 fr9n05.mhpcc.edu Avail 0 0 Idle 0 0.77 9999 R6000 AIX41 fr9n05.mhpcc.edu Avail 0 0 Busy 1 1.01 9999 R6000 AIX41 . . . . . . . . . . . . . . . . . . . . fr28n15.mhpcc.edu Avail 0 0 Busy 1 1.01 9999 R6000 AIX41 fr28n16.mhpcc.edu Avail 0 0 Busy 1 1.01 9999 R6000 AIX41 R6000/AIX41 217 machines 43 jobs 198 running 217 machines 43 jobs 198 running The Central Manager is defined on cws1.class.mhpcc.edu All machines on the machine_list are present

Use the -l option for a long listing. For example:
```
    llstatus -l 
```
By default, llstatus -l will provide a long list of all machines. This can be piped to grep to obtain useful information. For example, the command below will provide a list of all machines and their associated memory configurations.
```
    llstatus -l |grep -E "Machine|Memory"
```
If a particular machine name is specified, only the status of that machine is reported. For example:
```
    llstatus -l mymachine
```
See the llstatus man page for details
With xloadl, machine status information is displayed for all machines by default in the "Machines" window

Basic LoadLeveler Tasks

Canceling a Job

Use the llcancel command to cancel either running or queued jobs
Options available to cancel jobs by userid,hostname, queue host, cluster id or process id
Only the LoadLeveler System Administrator can cancel other users' jobs
For example, to cancel job mymachine.23.0:
```
    llcancel mymachine.23.0
```
See the llcancel man page for details
With xloadl, jobs can be cancelled by using the "Actions" menu from the "Jobs" window

Basic LoadLeveler Tasks

Displaying the Central Manager

The name of the Central Manager machine is shown at the end of the llstatus list.
With xloadl, the Central Manager can be found by using the "Actions" menu on the "Machines" window

Submitting Multiple Jobs

A single job command file can be used to submit multiple jobs
Accomplished by using multiple #@ queue statements
LoadLeveler statements in effect for the first job are generally in effect for all subsequent jobs in the same job command file
Can be useful if the same executable is to be run with different input and output files
The LoadLeveler System Administrator can set a limit the number of jobs any user can run at one time

This example job command file queues two jobs having different input, output and error files


#@ executable  = longjob
#
#@ input      = longjob.in1
#@ output     = longjob.out1
#@ error      = longjob.err1
#@ account_no = ABCDE-1234-567
#@ queue
#
#@ input      = longjob.in2
#@ output     = longjob.out2
#@ error      = longjob.err2
#@ account_no = ABCDE-1234-567
#@ queue

This example demonstrates the use of predefined LoadLeveler macros to generate different output files. Five jobs will be queued, each of which reads a unique input file and creates unique output and error files.


#@ executable = longjob
#
#@ input      = longjob.in.$(Process)
#@ output     = longjob.out.$(Cluster).$(Process)
#@ error      = longjob.err.$(Cluster).$(Process)
#@ account_no = ABCDE-1234-567
#@ queue
#@ queue
#@ queue
#@ queue
#@ queue

Using the Job Command File as the Executable

Useful when you need to do pre-execution job preparation and post-execution cleanup
Required for parallel jobs
Two ways to accomplish
- Do not use the #@ executable statement LoadLeveler will assume that the job command file is the executable.
- Use the name of your job command file with the #@ executable statement
No conflict - the shell considers LoadLeveler statements as comments because they begin with "#". LoadLeveler ignores all statements that do not begin with "#@"

In this example, the user's input file is copied to scratch space on executing host before execution. This saves the performance cost of accessing the input file over the network (local disk is more efficient). The job is then run, output copied back and finally, cleanup of the scratchspace is performed.


#!/bin/csh
#
# Beginning of LoadLeveler commands
#@ initialdir = /u/jsmith/LoadLeveler
#@ error      = run1.$(Cluster).err
#@ output     = run1.$(Cluster).out
#@ environment = MP_SHARED_MEMORY=yes
#@ account_no = ABCDE-1234-567
#@ queue

# Beginning of script commands

echo 'Copying input file to /localscratch'
cp input.1  /localscratch/input.1

echo 'Running the program'
run1

echo 'Copying output file back'
cp /localscratch/output.1   output.1

rm /localscratch/input.1
echo 'Cleanup done. Job completed.'
end

Submitting Parallel Jobs - General Notes

The ability to execute parallel jobs with LoadLeveler depends whether or not your system hardware/software supports parallel executions and how the LoadLeveler System Administrator has configured your system.

#@ job_type statement specifies type of job:


    #@ job_type = serial     -default 
    #@ job_type = parallel   -MPI

#@ node indicates the number of nodes requested for this job:
```
    #@ node = 2
```
#@ tasks_per_node indicates the number of tasks on each node. For Tempest #@ tasks_per_node should be 4:
```
    #@ tasks_per_node = 4
```
#@ total_tasks indicates the total number of tasks requested for the run. This can be used with either #@ node or #@ tasks_per_node , but not both, to indicate the total number of tasks:
```
    #@ total_tasks = 4
```
#@ network.MPI indicates which switch adapter is used, whether or not the adapter is shared, and which communications library is used. It is standard to use the switch adapter (css0), in a dedicated mode (not_shared) and with the User Space library (US):
```
    #@ network.MPI = css0,not_shared,US  
```
#@ environment statement is used for miscellaneous environment variables. Some of the more commonly used variables are:
```
    #@ environment = MP_SHARED_MEMORY=yes;  MP_INFOLEVEL=5; MP_MP_SAVEHOSTFILE=myhosts.txt;   
```
Parallel jobs can not be checkpointed
You can determine which nodes were used for your parallel execution in several ways. In your LoadLeveler job command file:
- Output the value of the LoadLeveler environment variable LOADL_PROCESSOR_LIST to a file which you can later view interactively. If you do not specify a file, the output will simply appear in your LoadLeveler output file. For example:
```
    echo $LOADL_PROCESSOR_LIST > myhosts
  
```
- Specify that mail be sent to you. It will automatically include the nodes used.
```
    #@ notification = complete
    #@ notify_user = jsmith@mhpcc.hpc.mil
  
```
- Set the MP_INFOLEVEL environment variable to a value above 1. Then examine the file where std.err is specified to be written.
```
    #@ error = myjob.err
    #@ environment = MP_INFOLEVEL=2
  
```

Pre-execution setup and post-execution cleanup is possible if the executing nodes are known. The example below demonstrates one way of accomplishing both.


#!/bin/csh
#@ initialdir        = /u/jsmith/LoadLeveler
#@ error             = run1.err
#@ output            = run1.out
#@ job_type          = parallel
#@ network.MPI       = css0,not_shared,US
#@ environment       = MP_SHARED_MEMORY=yes
#@ node              = 4
#@ tasks_per_node    = 4
#@ wall_clock_limit  = 12000
#@ account_no        = ABCDE-1234-567
#@ queue
#
set mydir  = "/u/jsmith/LoadLeveler/"
set infile = "input.1"
set nodes  = `echo $LOADL_PROCESSOR_LIST`

# Pre-execution setup
foreach node (${nodes})
  rcp ${mydir}${infile} ${node}:/localscratch
  echo "copied ${mydir}${infile} to ${node}:/localscratch"
end

run1

# Post-execution cleanup
foreach node (${nodes})
  rsh $node "cd /localscratch; rm -f ${infile}"
  echo "cleanup on ${node} done"
end

echo "Job completed"

Submitting Parallel MPI Jobs

When an MPI parallel code is executed, it contacts the SP Job Manager, which will attempt to allocate the number of nodes specified by the #@ node and #@ tasks_per_node statement.
Environment variables:
- It is important to specify #@ network.MPI = css0,not_shared,US if you want to use User Space protocol over the high performance switch. Without this, the slower default of IP communications will be used.
- A few other POE environent variables are able to be specified with the #@ environment statement, such as MP_INFOLEVEL and MP_LABELIO.
- Most other POE environment variables are ignored or overridden internally by LoadLeveler
- Be careful that you do not override your LoadLeveler command file environment variable settings with settings in your .cshrc, .login or .profile files. LoadLeveler "sources" these files when your job is started and any environment variable settings there will override what is in your job command file.

An example job command file for an MPI parallel execution appears below:


# Example MPI job command file
#
#@ initialdir      = /u/smith/proj01
#@ notification    = complete
#@ notify_user     = smith@favorite.machine.mail
#@ input           = myprog.in
#@ error           = myprog.$(Cluster).err
#@ output          = myprog.$(Cluster).out
#@ job_type        = parallel
#@ requirements    = network.MPI = css0,not_shared,US
#@ environment     = MP_INFOLEVEL=3;MP_LABELIO=yes;MP_SHARED_MEMORY=yes
#@ node              = 4
#@ tasks_per_node    = 4
#@ checkpoint      = no
#@ wall_clock_limit= 14000
#@ account_no      = ABCDE-1234-567
#@ queue

myprog

LoadLeveler Internals

Much of LoadLeveler's behavior is controlled by installation and configuration options set by the LoadLeveler System Administrator.
LoadLeveler configuration files are in the userid loadl home directory.
- The LoadL_admin file contains:
  - User stanzas - define characteristics and limits at a per user level
  - Class stanzas - define class parameters including resource limits, permissions,
  - Machine stanzas - define machine characteristics and if the machine is the Central Manager.
- The LoadL_config file contains global configuration information common to all machines across the pool, such as:
  - Job management policies
  - Which daemons to run on each machine
  - Which job classes are defined per machine
  - Job limits
  - Machine architectures
  - Job accounting parameters
  - Paths and environment variables
  - Time information
  - LoadLeveler macros
  - Other parameters too numerous to list here
Local configuration files can be created to tailor requirements for individual machines in the pool. Overide global configuration file.
LoadLeveler processes jobs and monitors the workload by running the following daemons (note that not all daemons run on all machines):

LoadL_master
The master daemon - manages all daemons on its resident machine
LoadL_schedd
The schedd daemon - manages batch submissions on its resident machine
LoadL_startd
The startd - accepts jobs for dispatch on its resident machine
LoadL_starter
The starter process - spawned by startd to manage a running job on the server machine
LoadL_shadow
The shadow process - spawned by the schedd to communicate with the starter process to run a job on the server machine
LoadL_kbdd
The keyboard daemon - monitors keyboard and mouse activity (AIX only)
LoadL_collector
The collector daemon - central collector of machine status from all machines in the pool
LoadL_negotiator
The negotiator daemon - the central scheduler and collector of job status from all machines in the pool
A typical (and slightly simplified) LoadLeveler job cycle might look like this:
1. User builds job command file
2. User submits job command file to LoadLeveler from local machine.
3. A LoadLeveler schedd daemon on the local machine communicates the job request to the negotiator daemon on the Central Manager.
4. The Central Manager determines when and where the job may run. Since the Central Manager knows about every machine in the pool, it can match job requirements with machine resources. After reaching this decision, the Central Manager sends a "permit to run" back to the schedd daemon. The Central Manager marks the job as "pending".
5. The originating schedd daemon spawns a shadow process to handle the job.
6. The shadow process contacts the startd daemon on the target machine where the job is assigned to run. It also establishes the necessary communication ports with this daemon.
7. The startd daemon on the target machine spawns a starter process which then receives the job information and executable from the shadow process. The starter process is responsible for running the user executable on the target machine. At this point, the shadow communicates to the Central Manager that the job has been started and its status is changed to "running".
8. After job completion, the starter process returns the exit status to the shadow , which then generates mail (optional), forwards the exit status to the schedd daemon and exits itself.
9. The schedd daemon finally notifies the Central Manager that the job has completed.

How LoadLeveler Schedules Parallel Jobs

Note: This section does not apply to MHPCC users. The MHPCC has implemented its own batch scheduler which supercedes the LoadLeveler scheduling mechanisms. Please see the "LoadLeveler at the MHPCC" section of this tutorial for details.

When you submit a job, you are required to specify the number of nodes needed by using the #@ node and #@ tasks_per_node keywords.
LoadLeveler will then use dynamic node allocation to assign nodes to your parallel job. You can not choose which nodes your job will run on by using a hostfile of any variety.
The job will run immediately if all of the following conditions are met:
- there are no other jobs queued at a higher system priority;
- you do not already have running the maximum number of jobs permitted to run simultaneously;
- there are enough nodes available.
If there is an insufficient number of nodes (such as when most nodes have been assigned to other jobs), the job is given a set amount of time to allocate its nodes.
- As the job allocates its nodes, no other jobs may use the nodes. They will appear to be "Idle" in the llstatus display, however, they are actually "allocating".
- If enough nodes are accumulated within the time allowed, the job will run.
- If the job fails to accumulate enough nodes, it is deferred (made inactive) for a set time interval. At the end of that time, it will attempt to accumulate nodes again. The job maintains its system priority (place in the queue) during the deferral period.
- The allocation / deferral parameters are site dependent and controlled by the LoadLeveler system administrator.
Users may submit as many jobs as they like, however this will have no effect on when their jobs will be run relative to other users if they exceed the maximum number of jobs permitted to run simultaneously (site configurable).

LoadLeveler at the MHPCC

Important: The MHPCC has implemented a batch scheduler which replaces most of LoadLeveler's scheduling mechanisms. Additionally, there are a number of site specific details unique to the MHPCC. Users should be certain to become familiar with this section before attempting to submit batch jobs on the MHPCC systems.

LoadLeveler Documentation: IBM's LoadLeveler manuals are available by using InfoExplorer.
MHPCC Scheduler Documentation: some general information is available from the MHPCC Technical Documentation WWW page. Most other documentation for users is included here. Curious users may also review the scheduler configuration file and docs located in /u/loadl/bqs, though most of this information will probably not be of much use for users.
Job Scheduling: The MHPCC scheduler is not a FIFO scheduler. Some jobs may "jump" ahead of others for various reasons including backfilling (see below) and job priority. There are many scheduling parameters which are configurable by the MHPCC and subject to change. Additionally, "special scheduling" is used to reserve nodes for a particular user or group. (Note: A full discussion of job scheduling is beyond the scope of this tutorial).
showq: This utility replaces the llq command. It lists the running, queued and non-queued jobs.
Path Variable: Include the LoadLeveler executables location in your path. For example, if your login shell is the C Shell, add the following line to your .cshrc file and then "source .cshrc" to make it take effect.
```
    set path=($path /usr/lpp/LoadL/full/bin)
  
```
Batch Classes: All batch nodes are configured as a single batch class. Users, therefore, do not need to specify a batch class in their LoadLeveler command file.
Number of Nodes per Job: Users may routinely specify anywhere from 1 to 128 nodes to run their job.
Account_no: The account_no keyword is mandatory at MHPCC. Your job WILL NOT be scheduled without a valid account number in your loadleveler command file, specific to your project. Please review account_no for further details.
Time Limits: The maximum amount of time permitted for a job to run is determined by the minimum number of nodes required for that job. The table below is current as of 8/8/97. For the most recent table, please see the MHPCC Configuration Summary.

Time Limit (hrs) Minimum# processors
8 65 - 128
16 33 - 64
24 5 - 32
36 1 - 4
Special Requests: Users who desire to run jobs with longer time limits or more nodes may submit a Special Request Form.
llsubmit : This command includes a front end screen which validates command files for a number of possible errors. Job commands file which are rejected will usually be accompanied by an error message stating why they were rejected.
llprio : Due to various constraints imposed by the scheduler backfilling feature, llprio may not be effective for users attempting to order their own jobs. The llhold command may be used instead.
Job Prioritization: Is entirely controlled by the MHPCC scheduler's configuration parameters and subject to change. In general:
1. Jobs are first screened for eligibility using site configurable parameters that define MHPCC scheduling approaches and fairness policies. Additionally, MHPCC system administrators are able to set higher/lower priorities for any given job.
2. Jobs are then passed along to a "feasibility pool," where they are prioritized based on job-related parameters such as the quality of service, size, length, and queue time.
3. Prioritized jobs are then scheduled for execution and assigned nodes on an immediate or future start time.
Backfilling: The MHPCC scheduler routinely runs smaller/shorter jobs ahead of larger/longer jobs if the available nodes would otherwise sit idle. One of three backfill algorithms may be used:
1. FIRSTFIT - Backfills jobs in the order it finds them in the queue.
2. BESTFIT - Selects the job from the queue that best fits the backfill window.
3. GREEDY - Identifies all combination of jobs that can run in the window, analyzing the quality of each. Jobs contained in the best schedule are backfilled.
Currently (8/8/97), the scheduler is using FIRSTFIT algorithm to backfill jobs. Users can take advantage of this by setting their wall_clock_limit keyword to the shortest amount of time required by their job.
showbf: Use this utility to view the current backfill "window".
wall_clock_limit: This keyword is required for a command script to be accepted by llsubmit at the MHPCC. This works to the user's advantage by allowing for a backfill capability.
Using large memory: AIX compilers have a maximum default of 256 MB for program data and stack size. To obtain more than this, you must do the following:
1. Compile your program with the -bmaxdata: option. For example, the following compile command will allocate 512 MB data segment:
```
 xlf -bmaxdata:512000000 -o myprog mprog.f  
```
  See the xlf man page for details about the -bmaxdata: option.
2. Make sure that the memory limits set in your shell are not too small. To do this, simply put the unlimit command in your .cshrc shell.
Examples. A few "getting started" examples are available in the LoadLeveler Exercise. Be sure to modify the job command files for your own use before attempting to run them in LoadLeveler. In particular:
- Use your actual userid and directories
- Use your actual email address
- Enable the wall_clock_limit statement (in seconds)
Number of Jobs Running and Queued: The MHPCC scheduler has parameters which control the number of jobs a user may have running and/or scheduled to be run. Be reasonable about the number of jobs you queue up - no more than 20 please. There is no advantage to queing more jobs as the MHPCC scheduler will not permit "queue stuffing".
Pathnames: When specifying your home directory in a job command file, do not use pathnames that begin with "/a" such as /a/raid2fr2sw/u8/jsmith. Instead, use something like /u/jsmith. Use of the "/a" paths will cause the command file to fail.
If you plan on using LoadLeveler's GUI, xloadl, add the xloadl X resource specifications to your .Xdefaults file. This step is optional, but if you're using xloadl, it makes it look a lot nicer.
1. Copy the xloadl X resource specification file to your own directory:
```
    cp $WORKSHOP/samples/loadl/Xdefaults.xloadl  .
  
```
2. Edit your .Xdefaults file to include the Xdefaults.xloadl file.
3. Make sure your DISPLAY variable and xhost permissions are set correctly.
4. Start xloadl with the command "xloadl &"

Time Limit (hrs)	Minimum# processors
8	65 - 128
16	33 - 64
24	5 - 32
36	1 - 4

Other helpful hints

If you have multiple #@ environment statements, only the last will have effect. If you need to specify multiple environment variables, separate them by semi-colons with a single #@ environment statment. For example:
```
#@ environment  = MP_Shared_MEMORY=yes;MP_INFOLEVEL=3;MP_LABELIO=yes
```
Don't forget to set up the network adapter and communications library for optimum performance:
```
#@ network.MPI  = css0,not_shared,US 
```
Communications throughput - rough estimates. You should expect something close to these for jobs with message sizes greater than 500,000 bytes in length.
- IP over ethernet = < 1 MB/sec
- IP over switch = 9-12 MB/sec
- US over switch = 30-34 MB/sec
- IP over new switch = 28-30 MB/sec
- US over new switch = 93-100 MB/sec
LoadLeveler will "source" both your .cshrc and .login files. C Shell users may wish to exclude LoadLeveler from running interactive only commands in these files by doing something like:
```
if ($?prompt) then
  setenv TERM vt100
  set filec
  set prompt =  "`hostname -s`% "
  setenv MP_EUILIB us
     :
     :
endif
```
Do not use the #@ executable statement if you are running parallel jobs. Parallel jobs use the job command file as the executable.
Use your FULL email address if you specify "notify_user" in your command script.
Do not try to use LoadLeveler macros, such as $(job_name), $(cluster) or $(process) as script variables. They will not be recognized by the shell.

LoadLeveler Job Command File Keywords Reference

An alphabetical list of the keywords you can use in a LoadLeveler job command file is provided below. All of these keywords are linked to their descriptions from the llsubmit man page.

account_no
arguments
checkpoint
class
comment
core_limit
cpu_limit
data_limit
dependency
environment
error
executable
file_limit
group
hold
image_size
initialdir
input
job_cpu_limit
job_name
job_type
max_processors
min_processors
notification
notify_user
output
parallel_path
preferences
queue
restart
requirements
rss_limit
shell
stack_limit
startdate
stepname
user_priority
wall_clock_limit

LoadLeveler Commands Reference

The following commands permit you to perform LoadLeveler related activities. Each is linked to its LoadLeveler man page.

llstatus: Shows you information about the machines in the LoadLeveler pool
llsubmit: Submits a LoadLeveler job command file for scheduling and execution
llq: Gets information about jobs
llcancel: Cancels a job
llhold: Holds/releases a job
llprio: Changes the priority of a job
xloadl: Invokes LoadLeveler's graphical user interface

References and More Information

IBM's LoadLeveler WWW page located at: http://www.rs6000.ibm.com/software/sp_products/loadlev.html
"IBM LoadLeveler User's Guide" IBM Corporation
"IBM LoadLeveler Administration and Planning Guide" IBM Corporation