1. What is the Parallel Operating Environment?
2. POE Definitions
3. Using POE: Overview
4. Understanding Your System Configuration
5. Establishing Authorization
6. Compiling and Linking a Parallel Program
7. Setting Up The Execution Environment
   1. Setting POE Environment Variables
   2. Basic POE Environment Variables
   3. Example Basic Environment Variable Settings
   4. Miscellaneous POE Environment Variables
8. Invoking the Executable
9. CPU and Communications Adapter Usage
10. Terminating a POE Job
11. Run-time Analysis Tools
12. Parallel File Copy Utilities
13. Site Specific Information and Recommendations
14. References and More Information
15. Appendix A: Programming Considerations
16. Exercise

 

What is the Parallel Operating Environment?

• The Parallel Operating Environment (POE) is part of IBM's Parallel Environment (PE) software product. From a user's perspective, and for discussion purposes, the terms POE and PE will be considered synonymous.

• POE encompasses a collection of software tools designed to provide an environment for developing, executing, debugging and profiling parallel C, C++ and Fortran programs.

• Some of POE's key components include:
  • Facilities to manage your parallel execution environment (environment variables and command line flags)
  • Message Passing Interface (MPI) library for interprocess communications
  • Parallel compiler scripts
  • X-Windows parallel run-time analysis tools
  • Parallel file copy utilities
  • Authentication utilities
  • Parallel debuggers
  • Visualization Tool (VT)
  • Parallel profiling tools

• Much of what POE does is designed to be transparent to the parallel user. Some of these tasks include:
  • Linking the necessary parallel libraries during compilation (via parallel compiler scripts)
  • Finding and acquiring machines (nodes) for your parallel job
  • Loading your executable onto all nodes acquired for your parallel job
  • Handling all stdin, stderr and stdout between the nodes of a your parallel job
  • Signal handling for all tasks in your job
  • Providing intertask communications facilities
  • Managing the use of processor and network adapter resources
  • Retrieving system and job status information when requested
  • Error detection and reporting
  • Providing support for run-time profiling and analysis tools

• POE is a distributed memory, message passing based system, designed to run on the IBM RS/6000 platform under AIX. POE software can be run on IBM SP machines, networked clusters of RS/6000 workstations, or a mix of the two. For discussion purposes, this tutorial will refer to "SP" systems, as these are used most commonly with POE.

• POE can also be used to run serial jobs and shell commands concurrently across a network of machines.

POE Definitions

Before learning how to use POE, understanding some basic definitions may be useful.

Node

For a parallel POE job, a node refers to single machine, which has a unique network name/address. There are a variety of IBM machine models which can be called a "node", including SMP machines that have multiple CPUs. For SMP machines, the entire machine is considered a node.

Pool

A pool is an arbitrary collection of nodes assigned by the system managers of an SP system. Pools are typically used to separate nodes into disjoint groups, each of which is used for specific purposes. For example, on a given system, some nodes may be designated as "login" nodes, while others are reserved for "batch" or "testing" use only.

Partition

The group of nodes on which you run your parallel program is called your partition. Typically, there are multiple active partitions for multiple users across an SP system. Depending upon how a given SP system is configured, the nodes in your partition may/may not be shared with other users' partitions.

Home Node / Remote Node

For interactive use, your home node is the node where you are logged into, and where you start your POE job. A Remote Node is any other non-home node in your partition. (Technically speaking, the home node may or may not not be considered part of your partition).

Job Manager

The Job Manager in POE version 2.4+ is a function provided by LoadLeveler, IBM's batch scheduling software. When you request nodes to run your parallel job, LoadLeveler will find and allocate nodes for your use. LoadLeveler also enables user jobs to take advantage of multiple CPU SMP nodes, and keeps track of how the switch is used for communications.

Partition Manager

The POE Partition Manager is a daemon process (poe) which is automatically started for you whenever you run an interactive POE job. The Partition Manager process resides on your Home Node, and is responsible for overseeing the parallel execution of your POE job. The Partition Manager generally operates transparently to the user, and is responsible for performing many of the tasks associated with POE:

• Obtains the nodes for your parallel job through communications with the Job Manager.
• Establishes socket connections with each of the nodes in your partition.
• Sets up your user environment and validation on each of the nodes in your partition.
• Starts a pmd daemon process on each of the nodes of your partition. This process serves as the point of contact between the Partition Manager and your task - it actually becomes the parent process of your executable.
• Dynamically links the specified communications library to your executable.
• Loads your executable from the home node to each node in your partition.
• Routes stdin, stderr and stdout on your "home node" to all other nodes in your partition.
• Exchanges control information (signaling, synchronization, exit status) with the pmd process on each node.

User Space Protocol

The fastest method for intertask MPI communications. Can only be used with the high performance switch. Often referred to simply as US protocol.

Internet Protocol

A slower, but more flexible method of intertask MPI communications. Can be used with other network adapters besides the high performance switch. Often referred to simply as IP protocol.

Non-Specific Node Allocation

Refers to the Job Manager (LoadLeveler)automatically selecting which nodes will be used to run your parallel job. Non-specific node allocation is usually the recommended (and default) method of node allocation. For batch jobs, this is typically the only method of node allocation available.

Specific Node Allocation

Enables the user to explicitly choose which nodes will be used to run a POE job. Requires the use of a "host list file", which contains the actual names of the nodes that must be used. Specific node allocation is only for interactive use, and recommended only when there is a reason for selecting specifc nodes.

Using POE: Overview

POE can be used both interactively and within a batch (scheduler) system to compile, load and run parallel jobs. The typical progression of steps is outlined below, and discussed in more detail in following sections.

1. Understand your system's configuration

2. Establish authorization on all nodes which you will use

3. Compile and link the program using one of the POE parallel compiler scripts

4. Set up your execution environment by setting the necessary POE environment variables. Create a host list file if using specific node allocation.

5. Start any run-time analysis tools (optional)

6. Invoke the executable

Understanding Your System Configuration

• Before running your parallel job, it is important to know a few details regarding the SP system you intend to use. This is especially important if you use multiple SP systems, as they will almost certainly be configured differently.

• Some of the questions you might ask:
  • How many nodes are available?
  • Are the nodes configured into separate pools?
  • What other jobs are running?
  • Are there sufficient machine resources (memory, disk) on the nodes I will be using?
  • Where can I run interactively? In batch?
  • How are other users using the system resources?

  Some hints/commands which may be used to answer these questions are discussed below.

• First, consult your site's documentation and/or support staff. In most cases, you should be able to find online information about the site's environment, policies, and system configuration.

• Next, find out which commands (if any) are available locally for you to query your system's configuration dynamically.

• Finally, take advantage of any of the standard IBM AIX or Parallel Environment commands which provide configuration information. Two example commands are shown below, each linked to its man page.

  llstatus: This LoadLeveler command has several options. If the -l (lowercase "L") option is used, a great amount of detailed information will be displayed for all nodes in the system. If this command is not in your path, try looking in your site's LoadLeveler directory, typically /usr/lpp/LoadL/full/bin/ or /usr/lpp/LoadL/bin/.

  lscfg: This AIX command should be available on all AIX systems. It can be used to obtain detailed configuration (memory, disk, adapters, etc.) information for a specific node. Note: you must be logged into, or have rsh access, to a node in order to use this command.

Establishing Authorization

• Establishing authorization in terms of POE means insuring that you are permitted to run parallel jobs on the nodes you intend to use.

• POE currently supports two types of user authorization:
  • AIX authorization (default)
  • DFS/DCE authorization

  Only AIX authorization will be covered here.

• AIX authorization can be setup by system administrators (root) in the /etc/hosts.equiv file, or by the user in a .rhosts file.

• If your system administrators have setup the /etc/hosts.equiv file, you can view it to determine which machines may be used, and possibly also, specifications on which users are permitted/denied access to which machines.

• If your system is not using the /etc/hosts.equiv file for authentication, then you will need to setup your own authentication file using the standard UNIX "trusted host" file, called .rhosts.

• Creating a .rhosts File

  • The format of the file is one machine name / userid per line. For example:

    hostname1  userid
    hostname2  userid
    hostname3  userid
    hostname4  userid
    ...
    

    where "hostname" is the actual name of an SP node and "userid" is your actual login userid on that node. If your userid is that same on all hosts, then you may omit it.

  • The file must be named .rhosts and you must create it in your home directory

  • It is recommended that you include the full domain name of all SP node names in your .rhosts file.

  • Make sure that your .rhosts file is writable only by you - AIX will invalidate it if it is writable by others. Making it readable only by you is suggested.

Compiling and Linking a Parallel Program

• AIX supports several "flavors" of C, C++ and Fortran serial and threaded compilers. POE provides parallel compiler "scripts" which automatically link in the necessary Parallel Environment libraries, and then call the appropriate serial or threaded AIX compiler.

• Both the native compilers and the parallel compiler scripts support a common command line syntax:

  [compiler]	[options]	[source_files]
  For example:
  mpxlf	-g -O3 -qlist -o myprog	mprog.f

• Compiler invocation commands: the table below summarizes which compiler "flavors" are typically available - although some may be dependent upon your site's license situation and software level. Note that all of the IBM compiler invocation commands are not shown. Other compiler commands are available to select IBM compiler extensions and features. Consult the appropriate IBM compiler user guides/references manual for details.

  IBM Compiler Invocation Commands
  Serial	xlc / cc	ANSI C compiler; subset of C++ compiler
  	xlC	C++ compiler
  	xlf / f77	Fortran 77 compatible; subset of Fortran 90 compiler
  	xlf90	Full Fortran 90 with IBM extensions
  Pthreads	xlc_r	xlc for use with pthreads programs
  	xlC_r	xlC for use with pthreads programs
  	xlf_r	xlf for use with pthreads programs
  	xlf90_r	xlf90 for use with pthreads programs
  MPI	mpcc	Compiler script for parallel C programs using MPI
  	mpCC	Compiler script for parallel C++ programs using MPI
  	mpxlf	Compiler script for parallel Fortran 77 programs using MPI
  MPI with Pthreads	mpcc_r	Parallel C compiler script for hybrid MPI/pthreads programs
  	mpCC_r	Parallel C++ compiler script for hybrid MPI/pthreads programs
  	mpxlf_r	Parallel Fortran 77 compiler script for hybrid MPI/pthreads programs
  HPF	xlhpf	High Performance Fortran; subset of xlhpf90
  	xlhpf90	High Performance Fortran 90

• Compiler options: all of the IBM compilers include many options - too numerous to be covered here. Some of these options are listed in the man pages hyperlinked to the compiler commands in the above table. For a full discussion, users are advised to consult the IBM documentation for details. An abbreviated summary of some of the more useful options are listed in the table below.

  Selected Common/Useful Compiler Options
  -bmaxdata:bytes -bmaxstack:bytes	Required for large memory use. Default data and stack (combined) size is 256 MB.
  -c	Compile only, producing a ".o" file. Does not link object files.
  -g	Produce information required by debuggers and some profiler tools
  -I (upper case i)	Names directories for additional include files.
  -L	Specifies pathname where additional libraries reside directories will be searched in the order of their occurrence on the command line.
  -l (lower case L)	Names additional libraries to be searched.
  -O -O2 -O3 -O4	Various levels of optimization
  -o	Specifies the name of the executable (a.out by default)
  -p -pg	Generate profiling support code
  -qarch=arch -qtune=arch	Permits maximum optimization for the SP processor architecture being used. Can significantly improve performance at the expense of portability.
  -qautodouble=setting	Automatic conversion of single precision to double precision, or double precision to extended precision
  -qlanglvl=level	Specifies which language standard (or superset or subset of a standard) to check against for nonconformance
  -qlist -qsource -qxref	Compiler listing/reporting options

• Note: All of the IBM compiler commands have default options, which can be configured by a site's system adminstrators. It may be useful to review the following files (if present) to learn exactly what the defaults are for the system you're using:

  /etc/*cfg*
  /usr/lpp/ppe.poe/lib/poe.cfg

• Linking: POE executables which use MPI or MPI with pthreads are dynamically linked with the appropriate communications library at run time. It is possible, but not recommended, to create statically linked modules. Please consult the "IBM AIX Parallel Environment for AIX: Operation and Use Volume 1" manual for details.

Setting Up The Execution Environment
Setting POE Environment Variables

• Setting up your execution environment primarily involves setting POE environment variables. There are approximately 40 POE environment variables that you can set to influence the operation of your execution environment.

• A complete discussion and list of the PE environment variables can be found in the "IBM AIX Parallel Environment for AIX: Operation and Use Volume 1" manual. They can also be reviewed (in less detail) in the POE man page.

• POE environment variables are set using either the setenv (csh/tcsh) or export (ksh, bsh) commands. For example:

  setenv MP_PROCS 64
  export MP_PROCS=64

  This can be done in several ways:

  • At the shell command prompt
  • Within your shell's "dot" files (.cshrc, .profile)
  • Within a script file which is "sourced" prior to execution

• All POE variables can be set/overridden with POE command line flags. See the POE man page for details.

Setting Up The Execution Environment
Basic POE Environment Variables

• Although there are many POE environment variables, you really only need to be familiar with a few of them. Specifically, those which answer the three basic questions:

  How many tasks / nodes do I require?

  MP_PROCS

  The number of task processes for your parallel job. May be used alone or in conjunction with MP_NODES and/or MP_TASKS_PER_NODE to specify how many tasks are loaded onto a physical SP node. The maximum value for MP_PROCS is dependent upon the version of PE software installed (currently ranges from 128 to 2048) If not set, the default is 1.

  MP_NODES

  Specifies the number of physical nodes on which to run the parallel tasks. May be used alone or in conjunction with MP_TASKS_PER_NODE and/or MP_PROCS. POE version 2.4+.

  MP_TASKS_PER_NODE

  Specifies the number of tasks to be run on each of the physical nodes. May be used in conjunction with MP_NODES and/or MP_PROCS. POE version 2.4+.

  How will nodes be allocated - should I choose them myself or let the LoadLeveler automatically choose them for me?

  MP_RESD

  Specifies whether or not LoadLeveler should be used to allocate nodes. Valid values are either "yes" (non-specific node allocation) or "no" (specific node allocation). If not set, the default value is context sensitive to other POE variables. Batch systems typically override/ignore user settings for this environment variable.

  MP_RMPOOL

  Specifies the SP system pool number that should be used for non-specific node allocation. This is only valid if you are using the LoadLeveler for non-specific node allocation (from a single pool) without a host list file. Batch systems typically override/ignore user settings for this environment variable.

  MP_HOSTFILE

  This environment variable is used only if you wish to explicitly select which nodes will be allocated for your POE job (specific node allocation). If you prefer to let LoadLeveler automatically allocate nodes then this variable should be set to NULL or "". If used, this variable specifies the name of a file which contains the actual machine (domain) names of nodes you wish to use. It can also be used to specify which pools should be used. The default filename is "host.list" in the current directory. You do not need to set this variable if your host list file is the default "host.list".

  Batch systems typically override/ignore user settings for this environment variable. Additional details on using the host list file are available here.

  Which communications protocol and network interface should I use?

  MP_EUILIB

  Specifies which of two protocols should be used for task communications. Valid values are either "ip" for Internet Protocol or "us" for User Space protocol. The default is "ip", while "us" is faster. Note: batch systems may differ in the default setting.

  MP_EUIDEVICE

  A node may be physically connected to different networks. This environment variable is used to specify which network adapter should be used for communications. Valid values are: "en0" (ethernet), "fi0" (FDDI), "tr0" (token-ring), or "css0" (high-performance switch). Note that valid values will also depend upon the actual physical network configuration of the node. For example, to specify "fd0" when a node does not physically have a FDDI adapter is an error. Recommendation: if all of your communication is between SP nodes, be sure to set MP_EUIDEVICE to css0.

Setting Up The Execution Environment
Example Basic Environment Variable Settings

• The following examples demonstrate how to set the basic POE environment variables for three different situations.

• Case 1: Non-specific node allocation of 4 tasks from a single pool with User Space protocol communications over the high performance switch.

  csh/tcsh	ksh/bsh
  setenv MP_PROCS 4 setenv MP_RMPOOL 0 setenv MP_RESD YES setenv MP_HOSTFILE "NULL" setenv MP_EUILIB us setenv MP_EUIDEVICE css0	export MP_PROCS=4 export MP_RMPOOL=0 export MP_RESD=YES export MP_HOSTFILE="NULL" export MP_EUILIB=us export MP_EUIDEVICE=css0

• Case 2: Non-specific node allocation using 4 physical nodes from a single pool. Each node will execute 4 tasks (for a total of 16 tasks), using User Space protocol communications over the high performance switch.

  csh/tcsh	ksh/bsh
  setenv MP_NODES 4 setenv MP_TASKS_PER_NODE 4 setenv MP_RMPOOL 2 setenv MP_RESD YES setenv MP_HOSTFILE "NULL" setenv MP_EUILIB us setenv MP_EUIDEVICE css0	export MP_NODES=4 export MP_TASKS_PER_NODE=4 export MP_RMPOOL=2 export MP_RESD=YES export MP_HOSTFILE="NULL" export MP_EUILIB=us export MP_EUIDEVICE=css0

• Case 3: Specific node allocation of 32 tasks (possibly from multiple pools) with Internet protocol communications over the high performance switch. Assumes the existence of a file called "myhosts" containing the desired hostnames.

  csh/tcsh	ksh/bsh
  setenv MP_PROCS 32 unsetenv MP_RMPOOL setenv MP_RESD no setenv MP_HOSTFILE myhosts setenv MP_EUILIB ip setenv MP_EUIDEVICE css0	export MP_PROCS=32 unset MP_RMPOOL export MP_RESD=no export MP_HOSTFILE=myhosts export MP_EUILIB=ip export MP_EUIDEVICE=css0

Setting Up The Execution Environment
Miscellaneous POE Environment Variables

A list of some commonly used, or potentially useful, POE environment variables appears below. A complete list of the POE environment variables can be viewed quickly in the POE man page. A much fuller discussion is available in the "IBM AIX Parallel Environment for AIX: Operation and Use Volume 1" manual.

MP_CPU_USE

Specifies whether or not your job is willing to share a node's CPU with other jobs. Valid values are "unique" and "multiple". The default for IP communications is "multiple". US communications defaults are "multiple" if specific node allocation is used, "unique" otherwise.

MP_ADAPTER_USE

Determines whether or not your job is willing to share a node's switch adapter with other jobs. Valid values are "dedicated" and "shared". The default is "shared" for IP communications, and "dedicated" for US communications.

MP_STDOUTMODE

Enables you to manage the STDOUT from your parallel tasks. If set to "unordered" all tasks write output data to STDOUT asynchronously. If set to "ordered" output data from each parallel task is written to its own buffer. Later, all buffers are flushed, in task order, to stdout. If a task id is specified, only the task indicated writes output data to stdout. The default is unordered. Warning: use "unordered" if your interactive program prompts for input - otherwise your prompts may not appear.

MP_INFOLEVEL

Determines the level of message reporting. Default is 1. Valid values are:

0 = error
1 = warning and error
2 = informational, warning, and error
3 = informational, warning, and error. Also reports diagnostic messages for use by the IBM Support Center.
3,4,5,6 = Informational, warning, and error. Also reports high- and low-level diagnostic messages for use by the IBM Support Center.

MP_LABELIO

Determines whether or not output from the parallel tasks are labeled by task id. Valid values are yes or no. If not set, the default is no.

MP_CHILD

Is an undocumented, "read-only" variable set by POE. Each task will have this variable set to equal its unique taskid (0 thru MP_PROCS-1). Can be queried in scripts or batch jobs to determine "who I am".

MP_RETRY

The period (in seconds) of time between processor node allocation retries if there are not enough processor nodes immediately available. Interactive use only.

MP_RETRYCOUNT

The number of times that the Partition Manager should attempt to allocate processor nodes before returning without running your program. Interactive use only.

MP_SAVEHOSTFILE

The name of an output host list file to be generated by the Partition Manager. Can be used to "save" the names of the hosts used by your POE job. Interactive use only.

MP_PGMMODEL

Determines the programming model you are using. Valid values are spmd or mpmd. If not set, the default is spmd. If set to "mpmd" you will be enabled to load different executables individually on the nodes of your partition.

MP_CMDFILE

Determines the name of a POE commands file used to load the nodes of your partition. If set, POE will read the commands file rather than STDIN. Valid values are any file specifier. Generally used only when MP_PGMMODEL=mpmd.

MP_EUIDEVELOP

Causes MPI to do more detailed checking during program execution. May be useful during program development. Valid values are "yes", "no", "debug" and "noc". The latter two settings turn on/off parameter checking. May significantly slow performance.

SP_NAME

Specifies the domain name of the Control Workstation. Can be used to select between different SP systems, each of which is managed independently.

• Note: not all POE variables are used by LoadLeveler. The following variables will have no effect if used in a LoadLeveler job command file. Please consult your site's documentation for other batch system restrictions or rules which may apply.

  MP_PROCS MP_NODES MP_TASKS_PER_NODE MP_RMPOOL MP_EUIDEVICE MP_HOSTFILE MP_SAVEHOSTFILE

  MP_PMDSUFFIX MP_RESD MP_RETRY MP_RETRYCOUNT MP_ADAPTER_USE MP_CPU_USE

Invoking the Executable

• Once the executable has been created, and the environment is setup, invoking the executable is relatively easy. It can be as simple as issuing the command: a.out

• The general syntax for invoking your executable is:

  [executable_name]	[POE_option_flags]	[executable_arguments]
  For example:
  myprog	-procs 3	arg1 arg2

• Command line flags may be used to temporarily override any POE environment variables that have been previously set. If used, an option flag is in effect for the duration of the program's execution only. See the POE man page for a complete listing of flags.

• For MPMD programs, the following steps must be performed:

• Set the MP_PGMMODEL environment variable to "mpmd". For example:

  C Shell: setenv MP_PGMMODEL mpmd
  Korn Shell: export MP_PGMMODEL=mpmd

• Enter poe at the Unix prompt. You will then be prompted to enter the executable which should be loaded on each node. The example below loads a "master" program on the first node and 4 "worker" tasks on the remaining four nodes.

  0:node1> master
  1:node2> worker
  2:node3> worker
  3:node4> worker
  4:node5> worker

• Alternatively, you can create a file which contains a list of the program names (one per line) which must be loaded onto your nodes. Set the environment variable MP_CMDFILE to that file name and poe will automatically load the nodes based upon that file when you invoke your executable.

• Execution starts automatically after the last node has been loaded.

For serial programs or commands, use the poe command followed by your program name or the command you wish to run across your partition. For example:

poe cp ~/input.file /tmp/input.file
poe my_serial_job
poe rm /tmp/input.file

 

CPU and Communications Adapter Usage

• Under POE version 2.4+, User Space jobs may have up to four tasks per node and up to 1024 tasks per job. IP jobs may have a maximum of 40 tasks per node and 2048 tasks per job.

The remainder of this section is intended primarily for interactive POE usage. SP batch systems, in general, do not permit users to share nodes, making the rest of the information in this section largely irrelevant.

• POE jobs typically require both CPU and communications adapter resources. The manner in which a job uses these two resources determines whether or not other users can run jobs on the same node.

• CPU Usage (MP_CPU_USE): Determines whether or not jobs from other users can run simultaneously on the same node(s) as your job. May be either "unique" or "multiple":

  • IP: default = multiple. Permits any number of other IP users and up to one other US user to run on the same node.

  • US specific node allocation: default = multiple. Permits any number of other IP users and NO other US users to run on the same node.

  • US non-specific node allocation: default = unique. No other users permitted on same node - either IP or US. Set the environment variable MP_CPU_USE to "multiple" to override this.

• Communications Adapter Usage (MP_ADAPTER_USE): Determines whether or not other user jobs may use the high performance switch adapter or not. May be either "shared" or "dedicated"

  • IP: default = shared. Permits any number of other IP users and up to one other US user to share the communications adapter on a node.

  • US: default = dedicated. Permits any number of other IP users and NO other US users to share the communications adapter on a node. Can not be overridden.

• Best performance is usually obtained running with US communications when CPU use is "unique" and the adapter is "dedicated".

• The best "good neighbor" policy is for all POE jobs to run with IP communications using the defaults of CPU use "multiple" and adapter "shared".

• For the most part, users do not need to change the default settings for CPU usage and communications adapter usage. One instance where the default might be changed concerns the use of US communications in an interactive pool of nodes shared by many users. In this case, it might be considered "good neighbor policy" to set MP_CPU_USE to multiple so that at least other IP jobs can run also.

 

Terminating a POE Job

• You can use CTRL-C to terminate a running, interactive POE job. This will send a SIGTERM signal to your home node's Partition Manager process, which in turn sends the termination signal to all remote tasks.

• It is highly recommended to hit CTRL-C only once. If it is hit more than once, it may kill the Partition Manager process before it completes the shutdown of your processes on the remote nodes. This can result in your processes continuing to run on many nodes, also tying up the switch adapter if User Space communications are being used. These nodes will be unavailable to other users since the LoadLeveler will think they are still in use by you.

• The poekill command can be used to safely terminate all of your POE tasks and daemons. The syntax of the command is:

  [poe | rsh]	poekill	[progname]	[poe_options]
  For example:
  poe	poekill	myprog	-procs 8 -hostfile myhosts
  rsh node12	poekill	myprog

  where myhosts is a host list file containing the names of the nodes where your tasks are running.

 

Run-time Analysis Tools

• The Parallel Environment software provides several X Windows based tools which can be used to analyze your parallel program's execution behavior. These tools are briefly described below. Please consult the "IBM AIX Parallel Environment for AIX: Operation and Use" manuals, Volume 1 and Volume 2 for detailed information.

• System Status Array
  • Lets you quickly survey the coarse utilization of processor nodes across your system.
  • This tool may be useful if you are not using LoadLeveler for node allocation and need to quickly find out which nodes are not being used.
  • May also be useful to obtain a quick and very coarse survey of how your job is using its nodes.
  • Invoked with the poestat command.

• Program Marker Array
  • This tool provides a grid of small squares, called lights, that change color under program control. It also provides for displaying task output strings and programmed light values.
  • Each task is represented by a "row" of lights. The number of lights in a row is determined by an environment variable.
  • Requires insertion of subroutine calls within an application's source code in order to effect light changes and output.
  • May be used to visually display the progress of a program's execution.
  • Source code is provided with the POE distribution software for users to borrow and freely modify.
  • Invoked with the pmarray command.

• Visualization Tool
  • The Visualization Tool (VT) is a graphical user interface which enables you to perform both:
    • Trace Visualization - post execution analysis of your application's behavior within IBM's Parallel Environment. Uses a trace log file produced during your program's execution.
    • Performance Monitoring - real time behavior of the SP system as your program (and other) programs execute
  • VT's primary purposes are to help you analyze, understand, debug and tune a parallel POE application by graphically demonstrating execution behaviors and resource utilizations.
  • VT provides over 30 graphical displays or "views" which present information about your application and the system. You, as the user, decide which views to display. Views fall into five major classes:
    • Computation (cpu use, load balance)
    • Communications (message passing events)
    • System (page faults, context switches)
    • Network (TCP/IP) activity
    • Disk activity
  • Most views can be "customized" and saved by the user.
  • Invoked with the vt command.
  • The tutorial is not available from the Maui High Performance Computing Center at this time.

• Xprofiler
  • This tool is a graphical version of the parallel grof command.
  • Permits profiling at the source line level
  • Provides all of the "flat" (non-graphical) gprof reports
  • Invoked with the xprofiler command.

 

Parallel File Copy Utilities

• POE provides four utilities which may be used to copy file(s) to and from a number of nodes.

• Three of these utilities (mcp, mcpscat, mcpgath) are actually message passing applications designed for efficiency. One utility (mprcp) is a shell script which uses the standard UNIX utility rcp to perform the copy operations and may not be as efficient.

  Note: see the associated hyperlinked man page for examples of each utility's use.

  Parallel File Copy Utilities
  mcp	Copies a single file from the home node to a number of remote nodes.
  mcpscat	Copies a number of files from task 0 and scatter them in sequence to all tasks, in a round robin order.
  mcpgath	Copies a number of files from all tasks back to task 0.
  mprcp	Copies a file from the home node to a list of remote hosts.

 

Site Specific Information and Recommendations

This section covers site specific details for POE usage at the MHPCC.

• Set MP_PROCS to the actual number of nodes you wish to use.

• Inform the Resource Manager about your job (MP_RESD = yes).

• Use dynamic linking of the communication libraries instead of static linking. This will give you more flexibility at run time - you can alternate between IP and US communications simply by changing the MP_EUILIB variable.

• Always use the high performance switch for SP communications (MP_EUIDEVICE = css0).

• Write scratch files to local scratch space whenever possible. At the MHPCC, each node has a /localscratch directory - it is faster than writing scratch files to your home directory.

• Interactive Users:

  • Try to use IP communications (MP_EUILIB = ip). This is "good neighbor" use of the limited number of interactive nodes. It will permit other POE jobs to run at the same time.

  • If you need (for testing purposes) to run with US communications in the Interactive pool, be sure to share the CPU with other users (MP_CPU_USE = multiple). If this is not done, even IP jobs can not run on the same node as your job.

  • Unless there is a particular reason for selecting certain nodes, do not use a host list file. Allow the Resource Manager to automatically allocate your nodes (MP_HOSTFILE = NULL or "").

  • During program development and debugging, it may be helpful to set MP_INFOLEVEL to a higher value (>3).

  • If you are having trouble getting the requested number of nodes for your job, be sure that you are using IP communications (MP_EUILIB = ip). If you still have problems, check what other users are doing in the Interactive pool with the command: jm_status -Pv | more . Most likely, somebody is running a User Space job without MP_CPU_USE set to multiple. Then notify the MHPCC User Services staff by sending email to "help@mhpcc.hpc.mil". (Note: the jm_status command is valid for POE versions prior to 2.4).

• LoadLeveler (batch) Users:

  • POE environment variables and adapter usage are specified in your LoadLeveler command file. For example:

    
        #@ requirements = (Adapter == "hps_user")
        #@ environment = MP_EUILIB=us;MP_INFOLEVEL=3;MP_LABELIO=yes
        

  • Always use US communication over the switch. The above example demonstrates how to do this.

  • Don't forget that certain POE environment variables in your "dot" files (.cshrc, .profile, .login) will override the settings you specify in LoadLeveler.

  • LoadLeveler always acquires nodes for POE jobs directly from the Resource Manager and ignores user host list files.

  • LoadLeveler ignores a number of POE environment variables. A list is available here.

 

References and More Information

• "IBM AIX Parallel Environment for AIX: Operation and Use Volume 1 Using the Parallel Operating Environment". IBM Corporation.
  www.austin.ibm.com/resource/aix_resource/sp_books/pe

• "IBM AIX Parallel Environment for AIX: Operation and Use Volume 2 Tools Reference". IBM Corporation.
  www.austin.ibm.com/resource/aix_resource/sp_books/pe

• "IBM AIX Parallel Environment for AIX: MPI Programming and Subroutine Reference". IBM Corporation.
  www.austin.ibm.com/resource/aix_resource/sp_books/pe

• "IBM AIX Parallel Environment for AIX: Installation". IBM Corporation.
  www.austin.ibm.com/resource/aix_resource/sp_books/pe

• "Parallel Operating Environment (POE)" tutorial from the Maui High Performance Computing Center. Permission granted to reproduce copyrighted material in whole or in part for United States Government or educational purposes.
  www.mhpcc.edu/training/workshop/poe.

• "Visualization Tool (VT)" tutorial from the Maui High Performance Computing Center. Permission granted to reproduce copyrighted material in whole or in part for United States Government or educational purposes.
  www.mhpcc.edu/training/workshop/vt.

• "IBM SP Parallel Environment (PE)" tutorial from the Cornell Theory Center.
  www.tc.cornell.edu/Edu/Talks/POE.

 

Appendix A: Programming Considerations

• POE presents several considerations for the application programmer, some of which can be very important. Programs which are unaware of these restrictions can behave unpredictably or fail unexplainably.

• Most of these considerations are covered in detail in the "IBM AIX Parallel Environment for AIX: MPI Programming and Subroutine Reference." manual. Only the most important ones are described below.

• The Parallel Environment supplies two different versions of the message passing libraries. Depending upon which parallel compiler script you use, your executables will be linked with either the signal-handling library (mpxlf, mpcc, mpCC) or with the threaded library (mpxlf_r, mpcc_r, mpCC_r).

• Developers should let POE handle signals when possible. All compiler scripts will link in POE's own signal handlers. These signal handlers include most signals that could cause program termination. POE does this so that it can terminate the entire parallel job in an orderly manner if one task terminates abnormally. It also does this so that it can complete certain pending tasks (such as the generation of VT trace files) before actually terminating.

• POE requires its own exit() and atexit() routines. Developers should not code their own.

• Passing string arguments: the shell will strip away single and double quotes used to delimit arguments. Quotes must therefore be "escaped" with backslashes. For example, the correct way to pass a string containing blanks as a single argument is:

  poe myprog \"this is one arg\"

• Large volumes of parallel stderr or stdout can overload the Partition Manager process on your home node (by exhausting IP buffers on the home node).

• For the MPI signal-handling library (mpcc, mpxlf, mpCC compiled programs), the POE signal handlers WILL cause a number of system routines to fail. Per IBM documentation:

  The message passing library uses an interval timer to manage message traffic, specifically to ensure that messages progress even when message passing calls are not being made. When this interval timer expires, a SIGALRM signal is sent to the program, interrupting whatever computation is in progress. The message passing library has a signal handler set, and normally handles the signal and returns to the user's program without the program's knowledge. However, the following library and system calls are interrupted and do not complete normally. The user is responsible for testing whether an interrupt occurred and recovering from the interrupt. In many cases, this is accomplished by just retrying the call.

  • sleep(see note below)/usleep/nsleep
  • select
  • open/close/fopen/fclose
  • pause
  • sigpause
  • accept
  • connect
  • recv/recvfrom/recvmsg
  • send/sendto/sendmsg
  • aio_read/aio_write/aio_suspend
  • fork
  • system
  • exec/execv/...
  • msem_lock/semop
  • AIX msg... routines
  • poll

• For the MPI threaded library, some considerations include:
  • Use of POE signal handlers
  • Default thread stack size of 64K bytes
  • Limitations on process forks
  • Standard I/O requires special attention
  • User is responsible for insuring that other linked libraries are thread safe.
  • The system include file <pthread.h> MUST be the first include file.
  • MPI_INIT must be called only once per task, not on each thread of that task. MPI_FINALIZE must be called on the same thread that called MPI_INIT.