Getting Started

RJA page needs formatting

Some basic information on using the Liverpool NW-GRID Beowulf clusters August 2008 Cliff Addison

Connecting to the Liverpool systems

The three machines currently on the Liverpool NW-GRID are:

Both lv1.nw-grid.ac.uk and lv3.nw-grid.ac.uk contain a mixture of dure-core and quad-core AMD Opteron processors. The quad-core processors are better known by their development name - Barcelona. The quad-core processors are able to run existing Opteron code, but performance may be enhanced on these nodes by recompiling codes with explicit Barcelona support. The main difference between the quad-core and dual-core nodes is that the quad-core nodes have 8 cores per node, with a minimum of 16 GBytes of memory rather than the 4 cores per node and 8 GBytes of memory found on the dual core processors.

lv2.nw-grid.ac.uk contains twin single-core Intel Xeon processors. Each node has 2 GBytes of memory.

Access is only possible using the Grid operations such as Condor-G, Globus or the gsissh family of commands (Grid Secure Shell). This note discusses using gsissh either from a grid-enabled Linux / UNIX system or from a Java enabled machine (Windows, Linux, Mac or other). The former is a command line interface that can be used within scripts to perform grid operations. The latter provides a windows-based access - more user friendly, but less flexible. Condor-G provides an extremely flexible way of running jobs on grid-enabled Liverpool systems if the necessary executables have already been installed on the system. Please see the note Obtaining Certificates to use with NW-GRID and other Globus configurations for information on obtaining a UK e- Science certificate and the Condor-G User's Guide for information on Condor-G. Obtaining and Using the Java-based GSI-SSHTerm

In order to run this application, the Java JDK 1.5 or higher has to be installed on your system. This is installed by default on many systems today, but if not, it can be downloaded and installed fairly easily.

Liverpool users with a Managed Windows Service system can install this by clicking on Start on the Windows Toolbar and then following the links

Install->Programming->Java 1.6.0_01

Other Windows and Linux users can go to the Java 5 download page and follow the links to download and install the latest version of J2SE JDK 5.0. Make certain that the full JDK is downloaded and installed and not just the smaller Run Time Environment (JRE). Note that the latest J2SE JDK 6.0. will also work and is now the recommended download.

Mac users should be able to get what they need (if it has not already been installed), from

http://www.apple.com/support/downloads/java2se50release1.html

Once you think you have the right Java installed, go to GSI-SSHTerm.

There is an option to load the Java Applet for GSI-SSHTerm and that will present you with what looks like a command window.

In a separate browser window, the instructions to use this command window can be found at GSI-SSHTerm Application Help. For reasons that are not entirely clear, there tend to be fewer problems if you authenticate using a certificate from your browser.

It is usually better and easier to run GSI-SSHTerm directly on your desktop. The easiest way to do this is to use Java Web Start. There is a link on the GSI-SSHTerm page. Clicking on this link not only sets up the GSI-SSH Term application, but also ensures that the latest version of GSI-SSHTerm is always used.

For more information on Java Web Start see

http://java.sun.com/products/javawebstart.

It is also possible to download a zip file of the latest version of this software. Once you have unpacked this file, you should see the top-level GSI-SSHTerm-X.YZ directory, (as of July 2008, X.YZ is 0.91d, but this changes frequently) under which you should see a bin and a lib sub-directories. Go into the bin directory. Linux / Mac users can run sshterm.sh.

A command window similar to the one in the browser should open on your desktop.

Once you have established a connection, several icons should become visible on the toolbar. In particular, the icon to start a secure FTP session will be visible. If you click on the SFTP icon, a separate window showing the home directory on the remote system will open. You can change to the directory you desire by double clicking on it or by typing the absolute path of the directory in the Address panel and hitting return.

Files can be added to this directory by clicking on File-> Upload Files and following the instructions. (Drag and drop also works under Windows).

s software. Once you have unpacked this file, you should see the top-level GSI-SSHTerm-X.YZ directory, (as of July 2008, X.YZ is 0.91d, but this changes frequently) under which you should see a bin and a lib sub-directories. Go into the bin directory. Linux / Mac users can run sshterm.sh.

A command window similar to the one in the browser should open on your desktop.

Once you have established a connection, several icons should become visible on the toolbar. In particular, the icon to start a secure FTP session will be visible. If you click on the SFTP icon, a separate window showing the home directory on the remote system will open. You can change to the directory you desire by double clicking on it or by typing the absolute path of the directory in the Address panel and hitting return.

Files can be added to this directory by clicking on File-> Upload Files and following the instructions. (Drag and drop also works under Windows).

Files can be downloaded from this directory by selecting the files of interest (click, ctrl-click or shift-click as per Windows systems) and then clicking on File->Download. If no files are selected, the entire directory is downloaded.

Users running X-windows on their desktop may be interested to know that it is possible to run X applications via GSI-SSHTerm. To ensure this is possible, in the "Connect to host..." window, click on the "Advanced" button. This opens the "Connection Profile" window. Click on the "X" tab and make certain that "Enable X Forwarding" is ticked. Click on the "Connect" button to continue with your connection.

One final point that is particularly relevant for users who are installing Java on their systems. The UK e-Science certificates can be too large to work with the default security arrangement under Java. To get around this problem, it is necessary to download and install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 5.0. Your systems admin may have already done this. However, if you get messages that contain phrases like

... - java.security.InvalidKeyException: Illegal key size

then it is likely that these policy files have not been installed. Fortunately, the procedure to do so is shorter than the name. Since these files are stored in the Java 5 SDK directory, the installation will have to be done by someone with write access to that directory.

The first stage is to down load the appropriate zip file. This can be found at the Java 5 download page, towards the bottom of the page under "Other Downloads". On the system where the Java is to be upgraded, unzip the downloaded file, currently with the name

jce_policy-1_5_0.zip.

This will create a directory jce, in which there is a README.txt file, two jar files and some other files. The README file provides instructions on where the jar files have to be copied. Once that has been done, the GSI-SSHTerm should work correctly, either from the zip file or Java Web Start. Command line gsissh operations

Grid secure shell (gsissh) commands are available on several CSD UNIX and Linux systems as well as on davinci.ssci.liv.ac.uk. Unfortunately, these commands also require some of the Globus client environment to be installed, so that installing the gsissh client software is not as simple as just downloading a precompiled tar file. At present, you need to modify your environment variables to use the Globus software:

You must have a valid proxy certificate before you use gsissh, or you will be prompted for a password, which you have not been given.

On systems, such as the above, that have a full Globus client installation, one way to generate proxy certificates is to first copy over your p12 certificate from your home computer to $HOME/.globus/usercred.p12 - this name and location is essential!

You then need to set the right file attributes:

The only copy of your certificate on that system should be the usercred file. Other users should not be able to access your certificate!

Once that is done, issue the command

[You'll be challenged for the certificate pass phrase]

This gives you a proxy lifetime of 12 hours. The notion of a proxy file merits comments. To quote from Ian Smith's Users Guide to Running Jobs on ULGrid using Condor-G,

A proxy certificate could be set up to last 48 hours, 30 minutes using:

You can check on the remaining validity period of your proxy certificate using the command:

When using gsissh, it is possible to establish an X-windows connection to the system if you use the command:

If you need to access lv1.nw-grid.ac.uk, lv3.nw-grid.ac.uk or lv2.nw-grid.ac.uk from a different system than those listed above, please contact one of the Liverpool e-Science team via helpdesk@liverpool.ac.uk.

Most users will be given a "grid" username of a form similar to lv1XY where XY are your two initials in your UK e-Science certificate. All of the gsissh operations will only work with that user account.

File transfers from Windows systems can be done via the GSI-SSHTerm (see above). In addition, command line companions to gsissh are

Once you are logged on, you are presented with a standard Linux environment. There are command line editors (vi, emacs), and if X-windows is supported on your client system, there are X-windows editors, such as X-emacs, as well as the nedit editor. More information on the latter can be found at http://www.nedit.org/documentation.php. Compilers and the like

lv3.nw-grid.ac.uk and lv1.nw-grid.ac.uk

As mentioned above, these systems use a mixture of quad-core and dual-core AMD Opteron processors. The dual-core nodes all have 8 GB of memory whilst the quad-core nodes have a mixture of 16 or 32 GB of memory. The 64-bit operating system deployed is currently SUSE 10.3.

The standard set of GNU compilers are available. The gcc, g++ and gfortran compilers are Version 4.2.1. The g77 compiler is 3.3.5 (the last g77 version).

The Portland suite of compilers (pgcc, pgCC, pgf77 and pgf95) are also available. As of July 2008, version 7.2 of this suite is installed. These compilers offer a richer set of compiler options and often produce much more efficient code for the AMD Opteron processor than the GNU compilers.

High performance libraries are often the only way to obtain high single processor performance. The system currently has the AMD Common Math libraries installed for the PGI compilers targeting either 32 bit or 64 bit binaries. These include the BLAS and LAPACK libraries within them. lv1 and lv3: For 64-bit binaries with the PGI compiler

BLAS and LAPACK

/opt/acml4.1.0/pgi64/lib/libacml.a

Go faster flags for the Portland compiler:

-fastsse -tp k8-64

On the quad-core (Barcelona) processors only, the recommended optimisation flags are:

-fastsse -tp barcelona-64

It must be stressed that this is just the initial offering of compilers and libraries, that different compilers (e.g. the Intel Fortran compiler) and different libraries (e.g. the Goto BLAS from Texas) can be installed if users have particular compiler or performance requirements. lv1 and lv3: For 32-bit binaries with the PGI compiler:

BLAS and LAPACK

/opt/acml3.6.0/pgi32/lib/libacml.a

General flags for the Portland compiler:

-tp k8-32 -Mcache_align

on both compile and link stages. Using -fast seems OK as a general optimisation flag, but I have not played with this extensively.

If BLAS and LAPACK libraries are needed for the GNU compiler, please let me know, as they can be made available.neral flags for the Portland compiler:

-tp k8-32 -Mcache_align

on both compile and link stages. Using -fast seems OK as a general optimisation flag, but I have not played with this extensively.

If BLAS and LAPACK libraries are needed for the GNU compiler, please let me know, as they can be made available. lv1 and lv3: Modules

Modules are now better integrated into Streamline's offering and are available on the head node and compute nodes by default.

You can see which modules are available using the command module avail

module avail

/usr/share/modules

3.1.6 modulefiles/mpich2 modulefiles/dot modulefiles/null modulefiles/intel modulefiles/pgi/7.1_linux86 modulefiles/module-cvs modulefiles/pgi/7.1_linux86-64 modulefiles/module-info modulefiles/pgi/7.2_linux86 modulefiles/modules modulefiles/pgi/7.2_linux86-64 modulefiles/mpich-ch-p4 modulefiles/score modulefiles/mpich-ch-p4mpd modulefiles/use.own

/usr/share/modules/modulefiles

dot null intel pgi/7.1_linux86 module-cvs pgi/7.1_linux86-64 module-info pgi/7.2_linux86 modules pgi/7.2_linux86-64(default) mpich-ch-p4 score mpich-ch-p4mpd use.own mpich2

/usr/local/share/modules/modulefiles

gcc/4.3 mpi/pgi-mpich2-64 globus/gt4.0.7(default) pcgamess6 mpi/SCore(default) pgi/7.1_linux86 mpi/gnu-mpich-32 pgi/7.1_linux86-64 mpi/gnu-openmpi-32 pgi/7.2_linux86 mpi/ifort-openmpi-64 pgi/7.2_linux86-64(default) mpi/pgi-mpich-64 sunstudio

To see loaded modules:

module list

To unload a module

module unload pgi/7.1_linux86-64 [or whatever the module name is.]

To load a module

module load pgi/7.1_linux86

lv2.nw-grid.ac.uk

This system uses Intel Xeon processors and a 32-bit operating system (currently Redhat Enterprise 4). Each node only has 2 GB of memory. There are 32 blades each with dual 3.0 GHz processors and 62 nodes each with dual 2.4 GHz processors.

The standard set of GNU compilers are available. The gcc, g++ and g77 compilers are Version 3.4.6.

The Intel Fortran compiler is available. As of July 2007, version 9.1 is installed. This is a full Fortran 95 compiler, with a set of compiler options to get good performance from the Xeon processor. This is the recommended Fortran compiler to use on this system.

The full Portland suite of compilers (pgcc, pgCC, pgf77 and pgf95) are also available. As of July 2008, version 7.2 of this suite is installed. These compilers were installed mainly to provide an alternative to the gcc compiler if this need arose.

High performance libraries are often the only way to obtain high single processor performance. The system currently has the Goto BLAS, LAPACK and the FFTW Version 3 FFT libraries compiled for the Intel compiler and the AMD Common Math libraries installed for the PGI compilers.

The Goto BLAS and LAPACK libraries can be found in /opt/intel/lib

When calling the BLAS / LAPACK routines from C, it is necessary to perform the final linking with the Intel compiler in order to access all of the run time libraries. This can be done using the ifort link options: -nofor_main -cxxlib-gcc along with any of your own link options and library references.

fftw3 libraries are found in /usr/local/fftw-3.0.1

The AMD Common Math Libraries are found under /opt/acml3.6.0/pgi32/lib or /opt/acml3.6.0/pgi32_mp/lib (for the multiprocessor libraries).

These include the BLAS and LAPACK libraries within them. Generally performance with the ACML BLAS is inferior to that obtained with the Goto BLAS. If the BLAS provide an important performance component of your application, you may want to consider using the Goto BLAS and the Intel compiler in preference to the PGI compiler and the ACML libraries.

As with the Goto BLAS, when the ACML BLAS are called from a C code, the final linking must be done with the PGI Fortran compiler using the link options -Mcache_align -Mnomain . Please note that the -Mcache_align compiler option must be used with either the Fortran and C compilers if the ACML library is called or segmentation faults will occur at run time. lv2: Modules

In your .bashrc file there should be the lines:

. /opt/modules/Modules/default/init/bash module load intel module load openmpi/intel32-gm

This sets-up your PATH etc. to access the Intel Fortran compiler and the associated OpenMPI wrappers and binaries. It also sets the BLASLIB environment variable for the absolute path to the Goto BLAS library.

You can see which modules are available using the command module avail

$ module avail

/opt/modules/Modules/versions

3.2.5

/opt/modules/Modules/3.2.5/modulefiles

dot openmpi/gnu32 intel/8.0 openmpi/gnu32-mx intel/9.1(default) openmpi/intel32-gm(default) module-cvs openmpi/pgi32-gm module-info pgi/7.0 modules use.own null

To see loaded modules:

module list

To unload a module

module unload intel/9.1 [or whatever the module name is.]

To load a module

module load pgi/7.0

Compiling and running parallel jobs

lv1.nw-grid.ac.uk

The necessary commands to run, compile and link codes, i.e. mpirun, mpicc, mpif90 etc should all be on your path from /usr/bin.

The syntax for the compiler wrappers is a little bit unusual.

For instance, to use the Portland Fortran 95 compiler with MPI, you need to specify

mpif90 -f90=pgf95 -mpif90_abi=pgi6

for both your compile and link stages. You would similarly need:

mpif77 -fc=g77 -mpif77_abi=gcc3

to use the g77 compiler,

mpicc -cc=gcc -mpicc_abi=gcc4

to use the gcc compiler and

mpicc -cc=pgcc -mpicc_abi=pgi6

to use the Portland C compiler.

Once users have a working executable, they are strongly urged to investigate submitting these jobs to lv1.nw-grid.ac.uk via Condor-G (see the Condor-G User's Guide for more information or contact an e-Science team member via the helpdesk). The section Example Programs should be viewed as a way to perform debugging or development runs. lv3.nw-grid.ac.uk

The necessary commands to run, compile and link codes, e.g. mpirun, mpicc, mpif90 etc should all be on your path from /opt/score/bin.

The syntax for the compiler wrappers means that the use of non-GNU (e.g. pgi or intel) needs to be specified.

For instance, to use the Portland Fortran 95 compiler with MPI, you need to specify

mpif90 -compiler pgi

for both your compile and link stages.

The default for mpicc, mpif77, mpif90 and mpic++ is to use the GNU compilers. Therefore to use the Portland C++ compiler, you also need to specify

mpic++ -compiler pgi

but to use the GNU g++ all that is needed is

mpic++

Once users have a working executable, they are strongly urged to investigate submitting these jobs to lv1.nw-grid.ac.uk via Condor-G (see the Condor-G User's Guide for more information or contact an e-Science team member via the helpdesk). The section Example Programs should be viewed as a way to perform debugging or development runs. lv2.nw-grid.ac.uk

The necessary commands to run, compile and link codes, e.g. mpirun, mpicc, mpif90 etc should all be on your path depending which openmpi modules have been loaded.

Notice that different compilers are only supported by loading the appropriate openmpi modules.

Once users have a working executable, they are strongly urged to investigate submitting these jobs to lv1.nw-grid.ac.uk via Condor-G (see the Condor-G User's Guide for more information or contact an e-Science team member via the helpdesk). The section Example Programs should be viewed as a way to perform debugging or development runs. Example Programs

In addition, in order to illustrate how parallel programs can be compiled, linked and then run under the Sun Grid Engine on this cluster example qsub scripts, test programs and a Makefile can be found in /usr/local/examples.

Most of the examples are based on a Fortran 90 demonstration program for pi that comes with the MPICH distribution.

It is probably simplest to copy this entire subdirectory to a subdirectory where you have write access. Full details (including some system dependent ones) are provided in the accompanying README file.

All the executables can be obtained by entering the command

make executable

where executable is the name of the desired code

I) Executing a sequential job interactively

The code sequential_pi can be formed and then run with the commands:

make sequential_pi

sequential_pi

This takes input from the keyboard.

NOTE: Running jobs on the main server is only acceptable for short test or debugging runs !

II) Executing a sequential job by submitting it to SGE

The script file qsub_sequential_pi.sh can be used to submit a job with sequential_pi to the batch queue, with input redirected from the file infile.

The necessary commands to perform this are:

make sequential_pi

qsub qsub_sequential_pi.sh

Two files will be generated after the job has run:

III) Submitting a parallel job using mpisub

The four codes that can be used with the mpisub command are: hello, pi3f90, cpi and pi_argument. The code job_launch is also available on lv3.nw-grid.ac.uk and lv2.nw-grid.ac.uk.

pi3f90 (and cpi) takes its input from the file infile redirected as standard input. pi_argument takes it input from the file infile whose name is passed as an argument. This file infile can be edited by the user if they so wish. It must terminate with a 0 or negative number.

To run this program on lv1.nw-grid.ac.uk with 8 cores over 2 nodes use:

mpisub 2x4 pi3f90 "< infile" To run this program on lv2.nw-grid.ac.uk with 8 processors over 4 nodes use:

mpisub 4x2 pi3f90 "< infile"

infile is a predefined file in this directory.

The mpisub command on either system will generate the file pi3f90.sh and will submit this script to the batch subsystem. The error and output files will have the form pi3f90.sh.e12345 and pi3f90.sh.o12345 respectively.

NOTE: The numbers at the end of these files correspond to the job number of the submitted batch job.

hello is a parallel variant of "Hello World" for multiple processes.

IV) Submitting a parallel job using qsub directly

The syntax involved in submitting a parallel job via qsub can easily be forgotten, so some examples that can provide a template for these jobs are provided.

V) Specifically accessing quad-core nodes via qsub - on lv1 and lv3 only

While mpisub will directly target the quad-core nodes if 8 cores per node are requested, users must specify a special resource if they want to ensure that their jobs submitted using qsub run only on the quad core processors. This can be done in one of two ways:

Example:

qsub -l quad qsub_pi3f90.sh

will use 2 of the quad-core nodes (with potentially 16 cores available). The number of cores used per node will depend upon the value specified within the job script, but normally users don't vary this quantity between runs.

NOTE: There are other ways in which the use of only quad-core nodes can be implemented. Only the simplest of these have been implemented to date, but other mechanisms can be provided (e.g. a quad-core only parallel environment) if users find such convenience helpful.

VI) lv3 and lv2 only - Submitting several sequential jobs as one parallel job

The last program, job_launch, source file mpi_job_launch.f, demonstrates one simple way to submit multiple sequential jobs as a single parallel one. This can be handy if you have to do lots of repeated runs or runs that are very similar to one another, but are independent.

This code assumes that each sequential run is specified in exactly one line of an input file. That line can contain multiple commands separate by semi-colons.

The command file used in this example is multi-pi, which contains:

./sequential_pi < inpi1 > out_pi1 ./sequential_pi < inpi2 > out_pi2 ./sequential_pi < inpi3 > out_pi3 ./sequential_pi < inpi4 > out_pi4

The number of lines in this file is used to control the number of processes used, so the file must not contain any blank lines at its end!

In this simple example, the sequential_pi program is run on several nodes at once with different input files on each run.

The parallel job does not complete until all of the sequential jobs have completed.

Further details about using the Sun Grid Engine, including details on the qsub job scripts with emphasis on the template script can be found at

SGE on lv3.nw-grid.ac.uk or SGE on lv2.nw-grid.ac.uk for the respective system.

LivStart (last edited 2009-02-12 15:52:50 by RobAllan)

This website maintained by Research Computing Services, University of Manchester