1. Login To The SP machine

   Make sure that you are logged into your assigned SP node with your assigned userid for this exercise. Ask the instructor if you have any questions.

2. Verify The Environment Variable $WORKSHOP

   The $WORKSHOP variable defines the root location for the workshop files, and may vary from workshop to workshop. Find out if this has already been setup for your workshop:

   echo $WORKSHOP

   If this environment variable is not set, check with the instructor for the correct location. Then, depending upon your shell, set $WORKSHOP:

   csh/tcsh	setenv WORKSHOP instructor/specified/path
   bsh/ksh	export WORKSHOP=instructor/specified/path

3. Create Your .rhosts File

   Make sure that you have a .rhosts file in your home directory. It should contain the name of every SP node which you may be using. If you have completed the POE exercises, this file may already be present. If not, use your favorite Unix editor to create it, or copy the example .rhosts file provided for the MHPCC SP training system:

   cp $WORKSHOP/samples/rhosts ~/.rhosts

4. Verify The Path For LoadLeveler Executables

   Make sure that the LoadLeveler commands are in your path. The following command should resolve the location of the llstatus command:

   which llstatus

   If the output from this command tells you it can't find llstatus, then follow the instructions in the table below. Upon completing them, the which llstatus command should work. If not, consult with the instructor.

   csh/tcsh

   Add the following line to your ~/.cshrc file. Be sure that your new path specification appears after all other path specifications. Also make sure that your .login file does not reset the path in conflict with the one shown below. set path=($path /usr/lpp/LoadL/full/bin /u/loadl/bin) After you have finished editing the file, issue the following command so that the changes have immediate effect. source ~/.cshrc

   bsh/ksh

   Add the following line to your ~/.profile file. Be sure that your new path specification appears after all other path specifications. export PATH=$PATH:/usr/lpp/LoadL/full/bin:/u/loadl/bin After you have finished editing the file, issue the following command so that the changes have immediate effect. . ~/.profile

5. Create A Subdirectory For The LoadLeveler Exercise Files

   mkdir ~/loadl
   cd ~/loadl

6. Copy The LoadLeveler Exercise Files

   The LoadLeveler exercise files include job command files, C and Fortran MPI source files and makefiles. Choose either the C or Fortran versions.

   C versions:	cp $WORKSHOP/loadleveler/samples/*.cmd ~/loadl cp $WORKSHOP/loadleveler/samples/*array*c ~/loadl cp $WORKSHOP/loadleveler/samples/*mpi_mm*c ~/loadl
   Fortran versions:	cp $WORKSHOP/loadleveler/samples/*.cmd ~/loadl cp $WORKSHOP/loadleveler/samples/*array*f ~/loadl cp $WORKSHOP/loadleveler/samples/*mpi_mm*f ~/loadl

   Make sure they are writable by you:

   chmod +w *

7. Run an MPI job with LoadLeveler

   1. First, create the MPI executables by using the makefiles. After successful execution of the commands below, you should have two executables called mpi_array and mpi_mm in your LoadLeveler subdirectory.

      C:	make -f make.mpi_array.c make -f make.mpi_mm.c
      Fortran:	make -f make.mpi_array.f make -f make.mpi_mm.f

   2. Review the LoadLeveler MPI example command files, mpi_array.cmd and mpi_mm.cmd and be sure you understand it before proceeding. Also be sure to change everything else as required, such as userid, paths, etc.

   3. Submit either/both of the LoadLeveler MPI example command files:

      llsubmit mpi_array.cmd
      llsubmit mpi_mm.cmd

   4. Immediately, issue the llq command to view your job in the LoadLeveler queue(s). This job will execute quickly, so you may miss it. You may also see the jobs of other users in the queues. Note: if you are using the MHPCC production system, you should use the showq command instead of llq.

   5. After a successful execution, you should find output files in your LoadLeveler subdirectory. Examine these files to determine if the program executed as expected.

      mpi_array.X.X.out - stdout and stderr from the mpi_array job
      mpi_array.hosts - list of hosts allocated for the mpi_array job
      mpi_mm.X.X.out - stdout and stderr from the mpi_mm job
      mpi_mm.hosts - list of hosts allocated for the mpi_mm job
      

8. Miscellaneous

Note for MHPCC production system users: the job ordering of LoadLeveler, including xloadl, is incorrect due to the MHPCC's batch scheduler. Therefore, steps 2 and 3 below should only be done during workshops and ignored for MHPCC production users.

1. Try using the llstatus command

2. Try using xloadl, the graphical user interface to LoadLeveler. There should be a file in your ~/loadl subdirectory called Xdefaults.xloadl. You can add the contents of this file to your ~/.Xdefaults file to improve the appearance of xloadl.

3. Modify one of the example LoadLeveler command scripts (*.cmd) so that it specifies a non-existent class, such as 18hr. Then submit the command script. Use the llq command or xloadl to monitor the job. It should hang in the queue. Use the llcancel command or xloadl to cancel the job.

This concludes the LoadLeveler exercises.