Installing and using Globus GT4.2.1 from the Source Distribution.

This page assumes you are installing Globus GT4.2.1 pre- Web services from the source distribution. It will make reference to the GROWL Scripts which were developed to simplify use of Grid resources. See http://www.growl.org.uk/Scripts/growl-scripts.shtml .

Client Installation and Setup

See http://www.growl.org.uk/Scripts/growl-scripts.shtml and follow the instructions. If there is a firewall at your site you may not be able to execute all the Grid commands from your desktop machine. For a table explaining the meaning of Globus error codes, see here http://www.nw-grid.ac.uk/GlobusHelp .

If you are logging onto a NW-GRID head node you can use that as a client. On the Daresbury cluster (dl1.nw-grid.ac.uk) we use the Panasas file system, so Globus is installed in /panfs/usr/local/globus-4.2.1. We have implemented an environment module for users as follows.

#%Module1.0###########################################################
# Module for Globus 4.2.1 pre-ws
######################################################################
proc ModulesHelp { } {
        puts stderr "\tAdds globus-4.2.1 pre-ws to your environment"
}

module-whatis    "Adds Globus 4.2.1 pre-ws to your environment"

setenv           GLOBUS_LOCATION /panfs/usr/local/globus-4.2.1
setenv           GLOBUS_TCP_PORT_RANGE 64000,65255
setenv           GLOBUS_HOSTNAME dl1.nw-grid.ac.uk

prepend-path     PATH /panfs/usr/local/globus-4.2.1/bin
prepend-path     LD_LIBRARY_PATH /panfs/usr/local/globus-4.2.1/lib
prepend-path     MANPATH /panfs/usr/local/globus-4.2.1/man

append-path      LIBPATH /panfs/usr/local/globus-4.2.1/lib
append-path      SHLIB_PATH /panfs/usr/local/globus-4.2.1/lib

Testing

To test that a Grid FTP server is working you can do something like the following.

grid-proxy-init
globus-url-copy -vb gsiftp://localhost/dev/zero file:///dev/null

This will run until killed.

Service Installation

For the full enchilada look here: http://www-unix.globus.org/toolkit/docs/latest-stable/admin/install/#gtadmin

1) Download and Compile

Firstly set up a user globus with group globus and log on as this user. The source can be built in this directory or in /panfs/usr/local/globus-4.2.1. Make sure you do not build the code in the same directory where the source is unpacked.

We also use the GROWL Scripts to install the server side middleware. If preferred, you can download the Globus source tarball from http://www-unix.globus.org/ftppub/gt4/4.2.1/installers/src/gt4.2.1-all-source-installer.tar.gz . Using GROWL will automatically download this (or a newer version) unpack it and make the relevant pre-ws components.

The Growl install will also apply any relevent patches in http://www-unix.globus.org/ftppub/gt4/4.2.1/updates/src (it sees what modules you have installed, checks the list on the above Web page, compares them and then installs the relevant ones) just building direct from the all-src-installer doesn't do that from the instructions below.

If doing it by hand the procedure is as follows:

tar xvf gt4.2.1-all-source-installer.tar.gz
cd gt4.2.1-all-source-installer
./configure --prefix=/panfs/usr/local/globus-4.2.1 --with-flavor=gcc32dbg
make common prewsgram gridftp gsi-openssh gsi-myproxy prews-test globus-gsi-test globus_ftp_client_test done
make install

On the configure step the string "gcc32dbg" is referred to as the "flavor". This is the commonest setting for a 32-bit architecture machine with the Gnu suite of compilers and debugging enabled. Note, the make step may take a long time. There may be errors produced at the end as we believe there are problems with the test components which can be omitted.

The client tools, as above, should now be in /panfs/usr/local/globus-2.4.1/bin.

If using Growl, install the GROWL Scripts then do the following, after editing the Growl/Makefile.conf for your local system settings.

cd Growl
make gt

This should result in a globus-4.2.1 sub-directory below the current directory. You can make your /panfs/usr/local/globus-4.2.1 a soft link to this location.

2) Install Certificates

Now, as root, create /etc/grid-security, obtain a host certificate for the certificate and installit as the file pair hostcert.pem and hostkey.pem with the mandatory access permissions (read only). Then copy to root and CA certificates that Growl has downloaded from globus-4.2.1/share/certificates into /etc/grid-security/certificates.

We assume you have a personal certificate for testing and can install it in an appropriate place.

3) Initial Testing

To set the required environment do

Growl/setup.sh

A first test can be done as follows:

globus-personal-gatekeeper -start
<string>
globus-job-submit <string> /bin/uname
globus-personal-gatekeeper -killall

Here, string is a handle returned by the command.

Basic Service Configuration

1) Ports

We have configured Globus and its associated services to use the ports as shown below. This is important for firewall configuration. Problems can be identified from the table of Globus error codes, see here http://www.nw-grid.ac.uk/GlobusHelp

2) /etc/services

Add the following lines to define the services that will run on these ports:

gsigatekeeper   2119/tcp                        # GSIGATEKEEPER
gsigatekeeper   2119/udp                        # GSIGATEKEEPER
gsissh          2222/tcp                        # GSI-SSH
gsissh          2222/udp                        # GSI-SSH
gsiftp          2811/tcp                        # GSI FTP
gsiftp          2811/udp                        # GSI FTP

3) GRAM using Xinetd

We use xinetd to manage the Globus gatekeeper GRAM service. You should have an /etc/xinetd.d directory and a a script /etc/init.d/xinetd to start the xinetd service. Then add a file /etc/xinetd.d/gsigatekeeper which matches the declaration in /etc/services with the following contents.

service gsigatekeeper
{
   socket_type  = stream
   protocol     = tcp
   wait         = no
   user         = root
   env          = LD_LIBRARY_PATH=/panfs/usr/local/globus-4.2.1/lib
   env += GLOBUS_HOSTNAME=dl1.nw-grid.ac.uk
   env += GLOBUS_TCP_PORT_RANGE=64000,65256
#   env += GRIDMAPDIR=/etc/grid-security/gridmapdir
   server       = /panfs/usr/local/globus-4.2.1/sbin/globus-gatekeeper
   server_args  = -conf /panfs/usr/local/globus-4.2.1/etc/globus-gatekeeper.conf
   instances = UNLIMITED
   disable      = no
}

When this file is added or modified in any way you will need to do '/etc/init.d/xinetd restart'.

The gatekeeper should now be listening on port 2119 and will be able to invoke a default fork jobmanager. A client can contact it and run an interactive job, but only if there is an appropriate line in the /etc/grid-security/grid-mapfile as follows. There will be one line per grid enabled user. We will consider pool accounts later.

"/C=UK/O=eScience/OU=CLRC/L=DL/CN=joe doe" dljxd

4) GSI-SSH using init.d

Installing GSI-SSH as root can be a bit scary, because you can lock yourself out of the system if it goes wrong. The Globus toolkit user guide at http://www.globus.org/toolkit/docs/4.2/4.2.1/security/openssh/admin recommends the following procedure. Start an alternative sshd server on a spare port by doing '/usr/sbin/sshd -p 2000 &' and log in via that port as a precution. However this is not strictly necessary as we will not be shutting down the original sshd server.

We recommend you keep the regular sshd for local users, so do not shut it down. You can add a new gsi enabled sshd running on port 2222 as follows. Before starting it you should check the SXXssd script to ensure it refers to this port on a line such as 'SSHD_ARGS="-p 2222"'.

cd /etc/init.d
cp /panfs/usr/local/globus-4.2.1/sbin/SXXsshd .
/etc/init.d/SXXsshd start

You should now test that gsi authentication works, e.g. from a remote machine with a valid proxy certificate by typing 'gsissh -p 2222 dl1.nw-grid.ac.uk'.

5) Grid FTP

gsiftp is set up in a similar way to GRAM above. A sample xined.d file grid-ftp is as follows.

service gsiftp
{
   instances = 1000
   socket_type  = stream
#   protocol     = tcp
   wait         = no
   user         = root
   env          = LD_LIBRARY_PATH=/panfs/usr/local/globus-4.2.1/lib
   env += GLOBUS_HOSTNAME=dl1.nw-grid.ac.uk
   env += GLOBUS_TCP_PORT_RANGE=64000,65256
#   env += GRIDMAPDIR=/etc/grid-security/gridmapdir
   server       = /panfs/usr/local/globus-4.2.1/sbin/globus-gridftp-server
   server_args  = -l -a -G /panfs/usr/local/globus-4.2.1
   nice = 10
   disable      = no
}

6) Other Components

Other components include MDS, MyProxy, BDII, GridWay, VOMS, etc. These are not currently installed on NW-GRID and will be documented only if users require them, e.g. for use with the NGS services.

Globus Job Managers

A range of jobmanagers, in addition to the default jobmanager-fork, can be configured so that they can be started by the Globus gatekeeper. We provide further information here.

In the GT4 release of Globus, two directories contain files associated with jobmanagers: globus-4.2.1/etc (the configuration files) and globus-4.2.1/lib/perl/Globus/GRAM/JobManager (the jobmanager Perl scripts). Jobmanagers scripts can be edited and multiple ones installed following the procedures below. The design of Globus allows jobmanagers to be created for specific purposes.

We will further document the jobmanager-sge configuration as it is done by hand rather than using scripts.

Unfortunately, at the time of writing, there seems to be no definitive set of jobmanager scripts. A number can be found on the Internet from diverse sources. They must be compatible with the schema used in the version of Globus installed (here 4.2.1) and should be checked for compatibility with the local system. They can be edited if necessary as indicated above.

Configuration Files

The default files include globus-gatekeeper.conf and globus-job-manager.conf. You should not need to edit these apart from making minor corrections and setting the desired log file location. If you add a new jobmanager, simply copy globus-job-manager.conf to a new one, e.g. globus-job-manager-sge.conf. Now look in the grid-services sub-directory where there should be a default jobmanager-fork script. This can be copied to a new one, such as jobmanager-sge and its contents edited. This is the script which is located by the globus-gatekeeper when it receives a request containing a string '2119/jobmanager-sge'. It might look as follows:

stderr_log,local_cred - /panfs/dl/home/globus/Growl/globus-4.2.1/libexec/globus-job-manager globus-job-manager -conf /panfs/dl/home/globus/Growl/globus-4.2.1/etc/globus-job-manager-sge.conf -type sge -machine-type unknown -publish-jobs

This tells the gatekeeper to launch libexec/globus-job-manager with the argument 'sge' using globus-job-manager-sge.conf.

Perl Scripts

libexec/globus-job-manager is an executable built when Globus is compiled. It should not be changed. The argument passed from the gatekeeper allows it to (indirectly) invoke a any number of jobmanager Perl scripts. Here we consider sge.pm as indicated by '-type sge' above. You could for instance create test.pm and change this to '-type test' for testing purposes.

Schema

Whilst strictly outwith the scope of these notes, the Globus jobmanager Perl scripts assume that data is passed to them via environment variables (attributes) and assume a number of states. These things are defined in .rvf files, known as validation files, which are located in globus-4.2.1/share/globus_gram_job_manager. The base globus-gram-job-manager.rvf can be extended, e.g. by adding sge.rvf into the same directory.

Please send corrections and comments on this page to Rob Allan.