1. Evolution of IBM's POWER Architectures
2. SP Hardware
   1. SP Frames
   2. SP Nodes
   3. High Performance Switch
   4. System Connectivity
   5. Control Workstation
   6. File/Install Servers
3. SP Software
   1. AIX Operating System
   2. System Administration
   3. Parallel Environment
   4. LoadLeveler
   5. SP Applications
4. Performance Benchmarks
5. References and More Information

Evolution of IBM's POWER Architectures

• In 1990, IBM announced the RISC System/6000 (RS/6000) family of superscalar workstations and servers based upon IBM's POWER architecture
  • POWER = Performance Optimized With Enhanced RISC
  • RISC = Reduced Instruction Set Computer
  • Superscalar = simultaneous execution of multiple instructions, including a combined floating point multiply-add (FMA) instruction which allowed a peak MFLOPS rate equal to two times the MHz rate.

• Initial configurations had a clock speed of 25 MHz, maximum data cache of 64 KB, single floating point and fixed point units, and a peak performance of 50 MFLOPS.

• RS/6000 Cluster: networked configuration of multiple RS/6000 machines. Provided initial basis for RS/6000 based, parallel, distributed computing.

SP1

• IBM's first SP (Scalable POWERparallel) system was the SP1. Essentially, it was a rack of standalone POWER1 processors connected by special software and network hardware. Its features included:
  • POWER1 processor with a 62.5 MHz clock with peak performance of 125 MFLOPS
  • Rack configuration - multiple machines (nodes) in a frame
  • High performance interprocessor communications over an internal switch network
  • System software for managing multiple machines
  • Parallel Environment software for parallel program developers

POWER2

• 1993: Continued improvements in the POWER1 processor architecture led to the POWER2 processor. Some of the POWER2 processor improvements included:

  • Dual floating point and fixed point units
  • Peak execution rate of 6 instructions per cycle
  • Increased data cache size
  • Increased memory to cache bandwidth
  • Clock speed of 66.5 MHz with peak performance of 254 MFLOPS
  • Improved instruction set (quad-word load/store, zero-cycle branches, hardware square root, etc)

SP2

• Improvements in the SP1 led to the SP2, which incorporated the POWER2 processor.

• Other improvements incorporated into the SP2 were directed at greater scalability and included improved system and application software and a higher bandwidth internal switch network for larger systems.

P2SC

• POWER2 Super Chip: An improved POWER2 processor made available in 1996 with clockspeeds increased up to a maximum of 160 MHz, which effectively doubled the performance of previous systems.

• Otherwise, virtually identical to POWER2 nodes, which it replaced.

PowerPC

• Introduced in 1993 as a result of the partnership between IBM, Apple and Motorola. The PowerPC 601 processor included most of the POWER instructions, but new instructions and features were added to support symmetric multiprocessors (SMP).

• Later models included the 603, 604 and finally, the 604e, which is a four-way SMP system running at 332 MHz. Currently, the 604e is the processor being used for the world's largest IBM SP system: ASCI Blue Pacific.

POWER3

• February, 1999: The most recent evolution of IBM's POWER architectures, the POWER3 processor combines the P2SC processor with the PowerPC architecture, and effectively replaces both in future evolutions of the RS/6000 line.

• 64-bit architecture. Initially, a 2-way SMP operating at a clock speed of 200 MHz.

• Significant improvements include:
  • Room for growth: fundamental design will be used for the next 3 generations of RS/6000 processors
  • Increased clock speed
  • 64-bit architecture
  • Dedicated integer and load/store units
  • Peak execution rate of 8 instructions per cycle (vs. 6 for the P2SC)
  • Cache improvements including an L2 cache of 1-16 MB.
  • Instruction and data prefetch

Futures: The Next Generations

• POWER3-II: Increase clock frequency 50% over POWER3 by tuning and moving into copper technology.

• POWER3-III: Increase clock frequency 40% over POWER3-II by tuning and moving into IBM's Silicon on Insulator (SOI) technology, for clockspeeds in the 500-600 MHz range.

• Increasing the number of SMP cpus within a POWER3 processor, to 8 and then to 16 cpus.

• Gigabyte internal switch network

SP Hardware
SP Frames

• An SP system is composed of 1 or more SP frames each containing multiple processor nodes

• Frame characteristics:

  • High Performance Switch internal network

  • Redundant frame power

  • Air cooled

  • Concurrent maintenance

  • 1 - 16 nodes per frame - depends on node types (thin/wide/high). Nodes fit into designated frame "slots".

  • Can mix different node types (wide/thin/high) within a frame

  • For larger systems (over 80 nodes), frames are used to house intermediate High Performance Switch hardware components
  
  

• Photos of an opened SP frame with 15 thin nodes and one empty slot:

  Front view

  Back view

  Click small image to see a larger image in a new window.

SP Hardware
SP Nodes

• A node in terms of the SP, is defined as a single, stand-alone machine, self-contained in a "box" which is mounted in an SP frame slot.

A node typically possesses the following characteristics: One of the POWER family of processors, such as a P2SC, 604e or POWER3 processor. Even though a node may be an SMP with multiple CPUs, it is still considered a single node. Its own independent I/O hardware, including disk drives. Its own independent network adapters, including an adapter for the internal switch network. Its own independent memory resources, including memory cards and caches. Its own copy of the AIX operating system. For SMP nodes, a single copy of the operating system is shared by all CPUs. Its own power and cooling equipment.

Inside view of a thin node Click for large image

• For the most part, SP nodes come in three different "packages":

  • Thin - occupies one "slot" in the frame.

  • Wide - occupies two horizontally adjacent slots in the frame. "Wideness" is due to hardware and space allowed for extra adapter expansion slots and disk/media bays.

  • High - occupies four adjacent (2x2) slots in the frame. Used for certain (older) types of SMP nodes.

• Currently, IBM is selling SPs with:

  • PowerPC (604e) thin and wide nodes

  • P2SC thin nodes,

  • POWER3 thin and wide nodes

• The table below summarizes configuration options for current SP processor types.

  Node Type	PowerPC 604e SMP	PowerPC 604e SMP	P2SC	POWER3 SMP	POWER3 SMP
  Packaging	Thin	Wide	Thin	Thin	Wide
  Processors per node	2/4	2/4	1	1/2	1/2
  Clock rates (MHz)	332	332	160	200	200
  Memory(min/max)	256 MB 3 GB	256 MB 3 GB	64 MB 1 GB	256 MB 4 GB	256 MB 4 GB
  L2 Cache (per processor)	256 KB	256 KB	n/a	4 MB	4 MB
  Slots available	2 PCI	10 PCI	4 MC	2 PCI	10 PCI
  Disk/Media bays	2	4	2	2	4
  Internal Disk (min/max GB)	0 36.4	0 72.8	4.5 18.2	0 36.4	0 72.8
  Bus speed: I/O adapters (MB/sec)	132	132	80	132	132/264
  Bus speed: Switch adapter (MB/sec)	400	400	160	480	480
  Switch Bandwidth (MB/sec)	160	160	115	160	160

• A table comparing older technology SP processor types is available here.

Processor Diagrams

POWER2/P2SC Processor Single CPU Superscalar, multiple-chip processor (separate functional units) Dual floating point and dual fixed point units Up to 4 memory cards with maximum memory-cache bandwidth needing 4 cards
POWER3 Processor SMP - multiple CPU cards Each CPU has its own data, instruction and L2 caches and access to the global shared memory L2 cache has its own bus - can be accessed simultaneously with main memory Separate data and address buses. Peak data throughput is 1.6 GB sec. I/O & switch bus operates concurrently and independently from memory-cache bus
POWER3 CPU Detail Superscalar, multiple-chip processor Dual floating point units Three fixed point units Two load/store units to sustain four floating point operations per cycle

 

SP Hardware
High Performance Switch

• The SP High Performance Switch provides the INTERNAL message passing fabric that connects all of the SP processors together into a single system.

• Topology: SP networks are bidirectional multistage interconnection networks (MIN's)

  • Bidirectional, any-to-any internode connection: allows all processors to send messages simultaneously

  • Multistage Interconnection: on larger systems (over 80 nodes), additional intermediate switches are added as the system is scaled upward.

Two basic hardware elements Switch board - One switch board per SP frame. Contains 8 logical switch chips with 16 physical chips for reliablility reasons. 8 logical chips wired as bidirectional 4-way to 4-way crossbar. Communications adapter - one adapter per SP node, which is cabled into a corresponding adapter in the node itself.	Photo
Sample 64 node switch configuration

• View additional switch configuration diagrams.

• SP High Performance Switch network characteristics:

  • Packet-switched network (versus circuit-switched)

  • Support for multi-user environment - multiple jobs may run simultaneously over the switch (one user does not monopolize switch)

  • Path redundancy - multiple routings between any two nodes. Permits routes to be generated even when there are faulty components in the system.

  • Built-in error detection

  • Hardware redundancy for reliability

  • Architected for expansion to 1000s of ports

  • Hardware latency: 500 ns up to 80 nodes, 875 ns for systems with up to 512 nodes

• Switch Adapters

  • TB1 - extinct?

  • TB2 - older Power2 systems

  • SPS - P2SC, older SMPs

  • SPSMX - 332 MHz SMPs

  • SPSMX2 - Power3 SMPs

• Switch Communication Protocols

  • Applications can use one of two available communications protocols:

    IP (Internet Protocol) - default; permits more tasks on the same node to share the adapter simultaneously with the tradeoff of lower bandwidth.

    US (User Space Protocol) - intended for parallel applications that require maximum communications performance. Limited number (1-4) of tasks on the same node which may use this protocol simultaneously.

  • Usage details are covered in the POE tutorial

• Switch Application Performance

  • An application's communication performance over the SP switch is dependent upon several factors:
    • Node type
    • Switch adapter type
    • Communications protocol used
    • Application specific characteristics

  • The table below shows performance metrics for an MPI message passing application under several different configurations. All results are from IBM documentation.

  Adapter Type	Communication Protocol	Node Type	Latency usec	Pt to Pt Bandwidth (MB/sec)
  TB2 (old)	IP	66 MHz Thin	312.1	9.9
  		66 MHz Wide	268.8	12.1
  	US	66 MHz Thin	40.0	35.4
  		66 MHz Wide	39.2	35.6
  SPS	IP	P2SC 120 MHz Thin		30.9
  		P2SC 160 MHz Thin
  	US	P2SC 120 MHz Thin	29.3	102
  		P2SC 160 MHz Thin	27.5	105
  SPSMX	IP	332 MHz SMP
  	US	332 MHz SMP	25	82
  SPSMX2	IP	200 MHz Power3	159	57
  	US	200 MHz Power3	22	133

SP Hardware
System Connectivity

• SP nodes are equipped with a varying number of adapter slots to permit a variety of I/O and network interfaces.

• Wide nodes have more slots than thin nodes

• One Ethernet LAN Standard

• Supported adapters:
  • Ethernet
  • FDDI
  • SCSI
  • BMCA
  • FCS
  • HIPPI
  • Token Ring
  • ESCON
  • ATM
  • High Performance Switch

SP Hardware
Control Workstation

• Serves as the single point of control for System Support Programs used by System Administrators for system monitoring, maintenance and control.

• Separate machine - not part of the SP frame

• Must be an RISC System/6000

• Connects to each frame with

  • RS-232 control line

  • external ethernet LAN

• Acts as install server for other SP nodes

• May also act as a file server (not recommended)

SP Hardware
File/Install Servers

• Typically, the full AIX operating system and IBM software is installed on all SP nodes.

• Upgrades to operating system or IBM software can first be installed on control workstation and then propagated to "install" nodes. Install nodes can, in turn, propagate upgrades to other nodes.

• SP nodes can be attached to auxillary disk to act as file servers. Wide nodes are usually used if this is desired.

• Other network connected machines can be used to serve files to SP nodes over the network.

• A simple SP configuration appears below

  

SP Software
AIX Operating System

• AIX/6000 is built upon UNIX System V and Berkeley Software Distribution 4.3 (4.3 BSD), with conformance to the Portable Operating System Interface for Computer Environments (POSIX) IEEE 1003.1-1990.

• Every SP node runs its own copy of the AIX operating system. SMP nodes have a single copy of the operating system which is threaded for all CPUs.

• AIX product information is available from IBM on the WWW at:

  • http://www.rs6000.ibm.com/software AIX General Information

  • http://www.rs6000.ibm.com/software/OS/aix43.html AIX 4.3

SP Software
System Administration

• SP system administration software includes a full set of tools used primarily by a System Administrator with "root" authority to perform the following tasks:

  • Install and configure the system

  • High Speed Switch installation and maintenance

  • Software installation

  • Manage user accounts

  • Manage file collections

  • Manage print and mail services

  • Manage automounter

  • Create diskless clients

  • Monitor and control system hardware. Examples of several monitoring tool displays:
    • Global Commands- control all nodes or a subset of nodes
    • All Node Temperatures- the temperatures of all nodes
    • All Node Power LED- the status of the power to all nodes
    • Frame Environment- the status of an SP frame
    • Node Front Panel- the logical front panel of a node

• Includes both proprietary and publically available code

SP Software
Parallel Environment

• The Parallel Environment (PE) is the SP software environment designed for the development and execution of parallel Fortran, C and C++ programs.

• PE consists of the following components

  • Parallel Operating Environment (POE)
    • Execution environment for parallel tasks
    • Interface to compilers

  • MPI Parallel Message Passing Library

  • Parallel Debuggers
    • pdbx
    • xpdbx

  • Programming Tools
    • Program Marker Array
    • System Status Array
    • Visualization Tool (VT)
      • Program Visualization
      • Performance Monitoring
    • Xgprof
    • Parallel prof and gprof

• Additional Information

  • POE Tutorial

  • VT Tutorial

  • MPI Tutorial

SP Software
LoadLeveler

• A batch job scheduling application

  • Schedules either serial or parallel jobs

  • Provides accounting information

  • Provides a graphical user interface for job submission and monitoring

  • Can be configured to interface with NQS system

• Runs on an SP and RS/6000, Sun, SGI and HP workstations

• Additional information:

  • LoadLeveler Tutorial

  • LoadLeveler Exercise

SP Software
Applications Software

• Over 10,000 AIX software applications are currently available.

• Detailed information for AIX software products can be accessed on IBM's WWW server at: www.austin.ibm.com/software/aix_appls.

  AIX Serial Application Areas
  Technical Architecture, Engineering, Construction Geographic Systems/Mapping Physical Sciences Scientific/Engineering	Mechanical Design and Analysis Fluids, CFD, Flow Analysis Solids and Drafting Structural Analysis, CSM
  General Business Accounting Legal Services Real Estate Distribution: Wholesale/Retail Industrial/Manufacturing Financial Services Health Care Insurance	Other Commercial Applications Education Government/Public Management Graphics Imaging and Document Management Mass Media/Communications Organizations/Fund Raising Public Utilities Transportation
  Cross Industry Artificial Intelligence Database Decision Support Information Management Integrated Workgroup Publishing Personal Information Managers Project Management	Development Tools Compilers/IDEs Cross Platform Development Tools Device Driver Editors Fourth Generation Languages On-Line Transaction Processing Systems Other Database Utility Software Design Tools Software Testing
  Networks and Communications	Internet and WWW Technologies

  AIX Parallel Applications
  Chemical and Pharmaceutical AMBER 4.0 AMPAC BATCHMIN 4.0 BIGSTRN-3 CHARMm 22 DGEOM DISCOVER 2.9 DISCOVER 2.9.5 DMOL 2.3 GAMESS(USA)	GAUSSIAN 94 HONDO 8.4 MOPAC 7.0 MULLEKIN NCSA Disco SIMBIG SPARTAN 3.0 WESDYN XPLOR 3.1
  Electronic Analysis ASX-P ATHENA BEBOP DAVINCI FAIM MCP2D MCP3D MEDICI NIAGRA	PISCES-MP PROCLAT PROCPHASE PWORDS SMART-SPICE STRIDE THUNDER VWF X-Wire
  Engineering Analysis and Research ABAQUS Airplane CFDS-FLOW3D DPAM/Cutter FIRE FLOW3P FLO67P Forge 3 GTNS2D LS-DYNA3D V930MP MARC(linear) MARC(non-linear) MSC/Eng. Application Code NEKTON	NPARC3D V1.0 PAM-CRASH PCG Solver POLYFLOW PVSOLVE RADIOSS RAMPANT RAYON SAMCEF-Structures SESTRA SPECTRUM STAR-CD SYSNOISE (Acoustic)
  Petroleum Exploration and Production COMP III COMP4 FOCUS 2D/3D ECLIPSE Family GEOVECTEUR MIGPACK MORE	OMEGA PROMAX SeisUP SimBest THERM VIP-EXECUTIVE
  Commercial and Database ACUCOBOL-85 ADABAS C ADSM/6000 ADVANTAGE/2000 CA-Unicenter CA/OpenIngres CICS/6000 DB2/6000 DB2/6000 Parallel Ed. DCE (Except DCE Manager) EDA/SQL ENCINA EpochBackup Client Epsilon Fed Fin Sys (FFS/6000) FOCUS HACMP 2.1 & HACMP 3.1 IDIS INFORMIX (OnLine DB) INFORMIX (DSA/xmp) Interactive Analysis System Job Scheduler for AIX Lawson Software LEGEND NATURAL NetWorker Neural Network Utilities NSL Unitree onGO OpenWorkFlow ORACLE 7.0 ORACLE 7.1	ORACLE Coop. Apps. Rel 10.4 PeopleSoft HRMS PeopleSoft Financials PeopleTools Performance Toolbox Prevail/XP-Jobtrac Remote Prevail/XP-Manager for RS/6000 Prevail/XP-PCS for RS/6000 PSF/6000 Pwrbnch COBOL Pwrbnch C++ Pwrbnch FORTRAN QUANTUM LEAP Red Brick Warehouse Red Brick Warehouse VPT REELlibrarian for AIX/6000 SAP R3 SAP R3 Parallel DB Support SAS System SequeLink SNAP SYBASE System 10 SYBASE Navigation Server System Management Template TME Triton TUXEDO Universal OLAS Vertical Market Solutions WorkFlow Template

Performance Benchmarks

NAS Parallel Benchmarks - Numerical Aerodynamic Simulation benchmark results maintained by the NASA Ames Research Center. The results below were taken from the November, 1997 report. Many other results are available on the NAS WWW site: www.nas.nasa.gov/NAS/NPB

Kernel Benchmarks EP (Embarrassingly Parallel) MG (Multigrid) CG (Conjugate Gradient) FT (3D FFT PDE) IS (Integer Sort)

Simulated CFD Applications LU (LU decompostion) SP (Scalar Pentadiagonal) BT (Block Tridiagonal)

SPEC Benchmarks maintained by the Standard Performance Evaluation Corporation located at: www.specbench.org

IBM SP performance specs located at: www.rs6000.ibm.com/hardware/largescale/SP/sp_specs.html

 

References and More Information

• "IBM RISC System/6000 Products & Services", IBM Austin WWW server.
  http://www.austin.ibm.com/

• "IBM Corporate WWW Server"
  http://www.ibm.com

• "IBM AIX Parallel Environment Operation and Use", IBM Corporation

• "IBM 9076 Scalable POWERparallel Systems Administration Guide", IBM Corporation

• IBM RS/6000 Scalable POWERparallel Systems Processor Node Comparison", IBM Corporation. June 19, 1995.

• High Performance Switch Overview. IBM Corporation. 1994.