Locked History Actions

GE/CreamCESetUp

Contents

  1. EMI_creamCE_GRID_Engine installation
    1. Interesting links
    2. GE installation links
    3. MPI installation links
    4. Preparing the installation and repositories
    5. Installing
      1. Installing the CREAM-CE software
      2. Installation of the batch system specific software
      3. Configure the GE batch system software before installing and configuring the CREAM software
      4. Installing MPI packages
      5. Configuration of a CREAM CE node in no cluster mode
      6. Configure the siteinfo.def file
      7. Sharing the GRID HOMES mounted front the EMC storage server with the WNs
        1. Configuring with YAIM (OK)
        2. Configuring IPTABLES on CreamCE
        3. Creating the SGE enviornment on the CreamCE
        4. Mounting the accounting file from SVG
        5. Configuring the queues and the File transfers
        6. Assigning complex to the jobs and modifying the sge_submit.sh script for it
        7. Configuring MPI
    6. BUGs/Known issues found after reconfiguring with YAIM
      1. Problem at first configuration with EMI-2 CREAM with GE
      2. Problem if there are "extra" characters in the beginning of the tomcat key file
      3. Bug on the sge_submit.sh script
      4. Removing sge ports from /etc/services
      5. It was necessary to load the SGE environment on the /var/lib/bdii/gip/plugin/glite-info-dynamic-ce script
      6. Modifying /var/lib/bdii/gip/ldif/ComputingShare.ldif
      7. Publishing the GlueCEPolicyMaxSlotsPerJob to run MPI jobs correctly
      8. Increase the value for the memory in /usr/bin/apel-sge-log-parser
      9. The BUpdaterSGE daemons does not inherits the correct GE environment variables
    7. Things to have in mind before reconfiguring with YAIM after some update
    8. TESTING
      1. Checking ldap (OK)
      2. Submitting a job from the CESGA UI
      3. Testing MPI jobs
        1. Applying 4 CPUS (OK)
        2. Applying 8 CPUS (OK)
        3. Second MPI job - Applying 4 CPUS (OK)
        4. jobOldSyntax - Applying 4 CPUS (OK)

EMI_creamCE_GRID_Engine installation

Preparing the installation and repositories

  • Install yum-priorities RPM and protectbase

yum install yum-priorities
yum install yum-protectbase.noarch 
  • Disabling yum auto-update

wget http://forge.cnaf.infn.it/frs/download.php/101/disable_yum.sh
sh -x disable_yum.sh 

Installing

  • Installing the CAs

yum install ca-policy-egi-core 
yum install lcg-CA

Installing the CREAM-CE software

  • Install the CREAM-CE metapackage:
  • yum install emi-cream-ce

====================================================================================================================================================================================================================
 Package                                                        Arch                                  Version                                                    Repository                                    Size
====================================================================================================================================================================================================================
Installing:
 emi-cream-ce                                                   x86_64                                1.1.0-4.sl6                                                UMD-2-updates                                2.5 k

[ .... ]

Installation of the batch system specific software

  • (!) READ the next advertisement if you are installing a CreamCe with GE

You have to install and configure the GE batch system software before installing and configuring the CREAM software. The CREAM CE integration was tested with GE 6.2u5 but it should work with any forked version of the original GE software. The support of the GE batch system software (or any of its forked versions) is out of the scope of this activity.

Before proceeding, please take note of the following remarks:

    CREAM CE must be installed in a separate node from the GE SERVER (GE QMASTER).
    CREAM CE must work as a GE submission host (use qconf -as <CE.MY.DOMAIN> in the GE QMASTER to set it up). 
  • After the installation of the CREAM CE metapackage it is necessary to install the batch system specific metapackage(s):
  • If you are running Torque, and your CREAM CE node is the torque master, install the emi-torque-server and emi-torque-utils metapackages:
  • If you are running GE, install the emi-ge-utils metapackage:

yum install emi-ge-utils
[ ... ]

====================================================================================================================================================================================================================
 Package                                                       Arch                                       Version                                           Repository                                         Size
====================================================================================================================================================================================================================
Installing:
 emi-ge-utils                                                  x86_64                                     2.0.0-1.sl6                                       UMD-2-updates                                     1.9 k
Installing for dependencies:
 glite-apel-sge                                                noarch                                     2.0.8-8.sl6                                       UMD-2-updates                                      24 k
 glite-info-dynamic-ge                                         noarch                                     6.0.0-4.sl6                                       UMD-2-updates                                      12 k
 glite-yaim-ge-utils                                           noarch                                     4.3.0-4.sl6                                       UMD-2-updates                                      17 k
 perl-XML-LibXML                                               x86_64                                     1:1.70-5.el6                                      sl                                                363 k
 perl-XML-NamespaceSupport                                     noarch                                     1.10-3.el6                                        sl                                                 16 k
 perl-XML-SAX                                                  noarch                                     0.96-7.el6                                        sl                                                 77 k
 perl-XML-Twig                                                 noarch                                     3.34-1.el6                                        sl                                                200 k

Transaction Summary
====================================================================================================================================================================================================================

Configure the GE batch system software before installing and configuring the CREAM software

  • In our case, we are using the GE batch system installed on SVG cluster, therefore, we need to use the same GE version in the CreamCE that in the SVG, so we have copied the GE installation directory from SVG to the CreamCE, and we have done the following steps in order to be able to execute GE commands from the CreamCE and be able to submit jobs to it:
  • Copying the GE directory

cd /usr/local
tar cvzf sge.tar.gz sge
scp -r sge.tar.gz root@cream.egi.cesga.es:/usr/local/
  • And on CreamCE side:

cd /usr/local;tar -zvxf sge.tar.gz 
  • Giving administrative and submission permissions to the new cream.egi.cesga.es on the system where is installed/configured the GE batch system.

(!)

[root@cream ~]# qconf -help | grep as
   [-as hostname_list]                      add a submit host
  
[root@cream ~]# qconf -help | grep ah
   [-ah hostname_list]                      add an administrative host
   [-ahgrp group]                           add new host group entry

qconf -as cream.egi.cesga.es
qconf -ah cream.egi.cesga.es 
  • Checking that the changes were done correctly:

[root@ce2 local]# qconf -ss | grep -i cream
cream.egi.cesga.es
[root@ce2 local]# qconf -sh | grep -i cream
cream.egi.cesga.es
  • Putting the new CreamCE in the /etc/hosts in the SVG in order to resolve to it:

[root@svgd ~]# grep -i cream /etc/hosts
84.21.169.124 cream.egi.cesga.es

Installing MPI packages

====================================================================================================================================================================================================================
 Package                                           Arch                                           Version                                               Repository                                             Size
====================================================================================================================================================================================================================
Installing:
 mpi-start                                         noarch                                         1.3.0-1.el6                                           UMD-2-updates                                          38 k

Transaction Summary
====================================================================================================================================================================================================================
  • yum install emi-mpi

====================================================================================================================================================================================================================
 Package                                               Arch                                          Version                                             Repository                                            Size
====================================================================================================================================================================================================================
Installing:
 emi-mpi                                               noarch                                        1.0.2-1.el6                                         UMD-2-updates                                        1.9 k
Installing for dependencies:
 glite-yaim-mpi                                        noarch                                        1.1.11-1.el6                                        UMD-2-updates                                         14 k

Transaction Summary
====================================================================================================================================================================================================================

Configuration of a CREAM CE node in no cluster mode

  • Make sure to place the two files in the target node into the /etc/grid-security directory. Then set the proper mode and ownerships doing:

chown root.root /etc/grid-security/hostcert.pem
chown root.root /etc/grid-security/hostkey.pem
chmod 600 /etc/grid-security/hostcert.pem
chmod 400 /etc/grid-security/hostkey.pem

Configure the siteinfo.def file

# YAIM example site configuration file - adapt it to your site!

MY_DOMAIN=egee.cesga.es

# Node names
# Note: - SE_HOST -->  Removed, see CLASSIC_HOST, DCACHE_ADMIN, DPM_HOST below
#       - REG_HOST --> There is only 1 central registry for the time being.
CE_HOST=cream.egi.cesga.es
LB_HOST=wms01.egi.cesga.es
WMS_HOST=wms01.egi.cesga.es
PX_HOST=myproxy.egi.cesga.es
BDII_HOST=topbdii.core.ibergrid.eu
BDII_LIST="topbdii.core.ibergrid.eu:2170"
MON_HOST=apel.egi.cesga.es
REG_HOST=lcgic01.gridpp.rl.ac.uk
SITE_BDII_HOST=sitebdii.egi.cesga.es

YAIM_LOGGING_LEVEL=DEBUG

#####################################
# CREAM CE node cluster mode  #
#####################################
## Be sure that CREAMCE_CLUSTER_MODE is set to no (or not set at all). 
## CREAMCE_CLUSTER_MODE="no"


#####################################
# CREAM CE Variables  #
#####################################
CEMON_HOST=cream.egi.cesga.es
CREAM_DB_USER=""
CREAM_DB_PASSWORD=""
MYSQL_PASSWORD=""
BLPARSER_HOST=cream.egi.cesga.es

##The CREAM sandbox directory name (default /var/cream_sandbox) can be changed using the yaim variable CREAM_SANDBOX_DIR". 
CREAM_SANDBOX_DIR="/home_grid/var/cream_sandbox"
CREAM_SANDBOX_PATH="/home_grid/var/cream_sandbox"


########################################
# SGE configuration variables #
########################################
BATCH_SERVER="svgd.cesga.es"
## Jobmanager specific settings
BLPARSER_WITH_UPDATER_NOTIFIER="true"
JOB_MANAGER="sge"
CE_BATCH_SYS="sge"
SGE_ROOT="/usr/local/sge/pro"
SGE_CELL="default"
## GE MASTER PORTS
SGE_QMASTER=6448
SGE_EXECD=6449
SGE_SPOOL_METH="classic"
SGE_CLUSTER_NAME="svgd" 
SGE_SHARED_INSTALL="yes"
BATCH_LOG_DIR="/usr/local/sge/pro/default/common"
BATCH_BIN_DIR="/usr/local/sge/pro/bin/lx26-amd64"
BATCH_VERSION="sge-6.2u5"


################################
# APEL configuration variables #
################################
APEL_MYSQL_HOST=apel.egi.cesga.es
APEL_DB_PASSWORD=""


################################
# MPI configuration variables #
################################
MPI_MPICH2_ENABLE="yes"
MPI_MPICH2_VERSION="1.3.2p1"
# If you do NOT provide a shared home, set $MPI_SHARED_HOME to "no" (default).
MPI_SHARED_HOME="yes"




# These variables tell YAIM where to find additional configuration files.
WN_LIST=/opt/glite/yaim/etc/wn-list.conf
USERS_CONF=/opt/glite/yaim/etc/users.conf
GROUPS_CONF=/opt/glite/yaim/etc/groups.conf
FUNCTIONS_DIR=/opt/glite/yaim/functions

# Set this to your prefered and firewall allowed port range
GLOBUS_TCP_PORT_RANGE="20000,25000"

# Site-wide settings 
SITE_EMAIL=grid-admin@cesga.es

#New yaim 3.0.1
SITE_SUPPORT_EMAIL=$SITE_EMAIL

SITE_NAME=CESGA
SITE_LOC="Santiago de Compostela, Spain"
SITE_LAT=42.875558          #42.8757 # -90 to 90 degrees
SITE_LONG=-8.553147         #-8.5536 # -180 to 180 degrees
SITE_WEB="http://www.egee.cesga.es"

# Architecture and enviroment specific settings
CE_CPU_MODEL=Opteron
CE_CPU_VENDOR=amd
CE_CPU_SPEED=2200

CE_OS="ScientificSL"    # Forma correcta
CE_OS_RELEASE=6.0
CE_OS_VERSION="Carbon"


#New variable
CE_PHYSCPU=30
CE_LOGCPU=720
CE_OS_ARCH=x86_64
CE_CAPABILITY="CPUScalingReferenceSI00=2395"
CE_OTHERDESCR="Cores=24,Benchmark=9.58-HEP-SPEC06"
SE_MOUNT_INFO_LIST="none"
CE_SI00=2395

CE_MINPHYSMEM=1524
CE_MINVIRTMEM=512
CE_SMPSIZE=2
CE_SF00=1714
CE_OUTBOUNDIP=TRUE
CE_INBOUNDIP=FALSE
CE_RUNTIMEENV="
    LCG-2
    LCG-2_1_0
    LCG-2_1_1
    LCG-2_2_0
    LCG-2_3_0
    LCG-2_3_1
    LCG-2_4_0
    LCG-2_5_0
    LCG-2_6_0
    LCG-2_7_0
    GLITE-3_0_0
    GLITE-3_0_2
    GLITE-3_1_0
    R-GMA
"

################################
# SE/DPM-MySQL configuration variables #
################################

# The name of the DPM head node 
DPM_HOST=se2.egi.cesga.es

# The DPM pool name
#DPMPOOL=the_dpm_pool_name
DPMPOOL=egee-pool

# The filesystems/partitions parts of the pool
#DPM_FILESYSTEMS="$DPM_HOST:/path1 my-dpm-poolnode.$MY_DOMAIN:/path2"
DPM_FILESYSTEMS="$DPM_HOST:/storage  $DPM_HOST:/mnt/xen01"

# The database user
#DPM_DB_USER=the-dpm-db-user
DPM_DB_USER=dpm_user

# The database user password
#DPM_DB_PASSWORD=the-dpm-db-user-pwd
DPM_DB_PASSWORD=

# The DPM database host
#DPM_DB_HOST=$DPM_HOST
DPM_DB_HOST=$DPM_HOST

# Specifies the default amount of space reserved  for a file
DPMFSIZE=200M

DPM_INFO_USER=dpm_info
DPM_INFO_PASS=

# This largely replaces CE_CLOSE_SE but it is a list of hostnames
SE_LIST="$DPM_HOST"
SE_ARCH="multidisk" # "disk, tape, multidisk, other"


## The following lines should not be neccesary, I just put them, but I think they are not used -- esfreire 16022012
#
# BDII/GIP specific settings
BDII_HTTP_URL="http://grid-deployment.web.cern.ch/grid-deployment/gis/lcg2-bdii/dteam/lcg2-all-sites.conf"
# Set this to use FCR
BDII_FCR="http://goc.grid-support.ac.uk/gridsite/bdii/BDII/www/bdii-update.ldif"
# Ex.: BDII_REGIONS="CE SE RB PX VOBOX"
# List of nodes included into CESGA SITE:
BDII_REGIONS="CE2 SE MON LFC WMS PX NAGIOS VOMS BDII TOPBDII"       # list of the services provided by the site
BDII_CE2_URL="ldap://$CE_HOST:2170/mds-vo-name=resource,o=grid"
BDII_SE_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid"
BDII_RB_URL="ldap://$RB_HOST:2135/mds-vo-name=local,o=grid"
BDII_MON_URL="ldap://$MON_HOST:2170/mds-vo-name=resource,o=grid"
BDII_PX_URL="ldap://$PX_HOST:2170/mds-vo-name=resource,o=grid"
BDII_LFC_URL="ldap://lfc.egi.cesga.es:2170/mds-vo-name=resource,o=grid"
BDII_WMS_URL="ldap://$WMS_HOST:2170/mds-vo-name=resource,o=grid"
BDII_FTS_URL="ldap://$FTS_HOST:2170/mds-vo-name=resource,o=grid"
BDII_NAGIOS_URL="ldap://rnagios.ibergrid.cesga.es:2170/mds-vo-name=resource,o=grid"
BDII_VOMS_URL="ldap://voms.egi.cesga.es:2170/mds-vo-name=resource,o=grid"
BDII_BDII_URL="ldap://sitebdii.egi.cesga.es:2170/mds-vo-name=resource,o=grid"
BDII_TOPBDII_URL="ldap://topbdii.egi.cesga.es:2170/mds-vo-name=resource,o=grid"

# VOS="atlas alice lhcb cms dteam biomed"
# Space separated list of supported VOs by your site
VOS="ops cesga dteam biomed compchem fusion imath.cesga.es ops.vo.ibergrid.eu iber.vo.ibergrid.eu tut.vo.ibergrid.eu eng.vo.ibergrid.eu ict.vo.ibergrid.eu life.vo.ibergrid.eu hpc.vo.ibergrid.eu earth.vo.ibergrid.
eu bing.vo.ibergrid.eu pfound.vo.ibergrid.eu phys.vo.ibergrid.eu social.vo.ibergrid.eu chem.vo.ibergrid.eu auger mpi"
QUEUES="GRIDEGI_ops GRIDEGI_large GRIDEGI_small GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_small GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_la
rge GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large"
VO_SW_DIR="/opt/exp_soft2"

#New in Yaim 3.0.1
GRIDEGI_OPS_GROUP_ENABLE="ops /VO=ops/GROUP=/ops/ROLE=lcgadmin"
GRIDEGI_SMALL_GROUP_ENABLE="dteam /VO=dteam/GROUP=/dteam/ROLE=lcgadmin ops.vo.ibergrid.eu /VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=VO-Admin /VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=Produ
ction"
GRIDEGI_LARGE_GROUP_ENABLE="cesga /VO=cesga/GROUP=/cesga/ROLE=VO-Admin biomed /VO=biomed/GROUP=/biomed/ROLE=lcgadmin  compchem /VO=compchem/GROUP=/compchem/ROLE=lcgadmin fusion /VO=fusion/GROUP=/fusion/ROLE=VO-ad
min imath.cesga.es /VO=imath.cesga.es/GROUP=/imath.cesga.es/ROLE=VO-Admin iber.vo.ibergrid.eu /VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/ROLE=VO-Admin /VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/RO
LE=Production tut.vo.ibergrid.eu /VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=VO-Admin /VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=Production eng.vo.ibergrid.eu /VO=eng.vo.ibergrid.eu/GROUP=/en
g.vo.ibergrid.eu/ROLE=VO-Admin /VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu/ROLE=Production ict.vo.ibergrid.eu /VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu/ROLE=VO-Admin /VO=ict.vo.ibergrid.eu/GROUP=/ict.
vo.ibergrid.eu/ROLE=Production life.vo.ibergrid.eu /VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=VO-Admin /VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=Production hpc.vo.ibergrid.eu /VO=hpc.vo
.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=VO-Admin /VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=Production earth.vo.ibergrid.eu /VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=VO-Admin /VO=ea
rth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=Production bing.vo.ibergrid.eu /VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=VO-Admin /VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=Producti
on pfound.vo.ibergrid.eu /VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=VO-Admin /VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=Production phys.vo.ibergrid.eu /VO=phys.vo.ibergrid.eu/GRO
UP=/phys.vo.ibergrid.eu/ROLE=VO-Admin /VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu/ROLE=Production social.vo.ibergrid.eu /VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=VO-Admin /VO=social.vo.ib
ergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=Production chem.vo.ibergrid.eu /VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=VO-Admin /VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=Production auger
 /VO=auger/GROUP=/auger/ROLE=SoftwareManager /VO=auger/GROUP=/auger/ROLE=Production /VO=auger/GROUP=/auger/FNAL mpi /VO=mpi/GROUP=/mpi/ROLE=SoftwareManager /VO=mpi/GROUP=/mpi/ROLE=Production"

# VO specific settings. For help see: https://lcg-sft.cern.ch/yaimtool/yaimtool.py
#::::::::::::::
#auger
#::::::::::::::
VO_AUGER_SW_DIR=$VO_SW_DIR/augersgm
VO_AUGER_DEFAULT_SE=$DPM_HOST
VO_AUGER_STORAGE_DIR=$CLASSIC_STORAGE_DIR/auger
VO_AUGER_QUEUES="GRIDEGI_large"
VO_AUGER_VOMS_SERVERS='vomss://voms1.egee.cesnet.cz:8443/voms/auger'
VO_AUGER_VOMSES="'auger voms1.egee.cesnet.cz 15004 /DC=org/DC=terena/DC=tcs/C=CZ/O=CESNET/CN=voms1.egee.cesnet.cz auger 24'"
VO_AUGER_VOMS_CA_DN="'/C=NL/O=TERENA/CN=TERENA eScience SSL CA'"


#::::::::::::::
#mpi 
#::::::::::::::
VO_MPI_SW_DIR=$VO_SW_DIR/mpisgm
VO_MPI_DEFAULT_SE=$DPM_HOST
VO_MPI_STORAGE_DIR=$CLASSIC_STORAGE_DIR/mpi
VO_MPI_QUEUES="GRIDEGI_large"
VO_MPI_VOMS_SERVERS='vomss://voms1.egee.cesnet.cz:8443/voms/mpi?/mpi'
VO_MPI_VOMSES="'mpi voms1.egee.cesnet.cz 15030 /DC=cz/DC=cesnet-ca/O=CESNET/CN=voms1.egee.cesnet.cz mpi 24'"
VO_MPI_VOMS_CA_DN="'/DC=cz/DC=cesnet-ca/O=CESNET CA/CN=CESNET CA 3'"
VO_MPI_WMS_HOSTS="wms1.egee.cesnet.cz wms2.egee.cesnet.cz" 


#::::::::::::::
#biomed
#::::::::::::::
VO_BIOMED_SW_DIR=$VO_SW_DIR/biomed
VO_BIOMED_DEFAULT_SE=$DPM_HOST
VO_BIOMED_STORAGE_DIR=$CLASSIC_STORAGE_DIR/biomed
VO_BIOMED_QUEUES="GRIDEGI_large"
VO_BIOMED_VOMS_SERVERS="'vomss://cclcgvomsli01.in2p3.fr:8443/voms/biomed'"
VO_BIOMED_VOMSES="'biomed cclcgvomsli01.in2p3.fr 15000 /O=GRID-FR/C=FR/O=CNRS/OU=CC-IN2P3/CN=cclcgvomsli01.in2p3.fr biomed'"
VO_BIOMED_VOMS_CA_DN="'/C=FR/O=CNRS/CN=GRID2-FR'"


#::::::::::::::
#cesga
#::::::::::::::
VO_CESGA_SW_DIR=$VO_SW_DIR/cesga
VO_CESGA_DEFAULT_SE=$DPM_HOST
VO_CESGA_STORAGE_DIR=$CLASSIC_STORAGE_DIR/cesga
VO_CESGA_QUEUES="GRIDEGI_large"
VO_CESGA_USERS=vomss://voms.egi.cesga.es:8443/voms/cesga?/cesga
VO_CESGA_VOMS_SERVERS="'vomss://voms.egi.cesga.es:8443/voms/cesga?/cesga'"
VO_CESGA_VOMSES="'cesga voms.egi.cesga.es 15110 /DC=es/DC=irisgrid/O=cesga/CN=host/voms.egi.cesga.es cesga'"
VO_CESGA_VOMS_CA_DN="'/DC=es/DC=irisgrid/CN=IRISGridCA'"


#::::::::::::::
#compchem
#::::::::::::::
VO_COMPCHEM_SW_DIR=$VO_SW_DIR/compchem
VO_COMPCHEM_DEFAULT_SE=$DPM_HOST
VO_COMPCHEM_STORAGE_DIR=$CLASSIC_STORAGE_DIR/compchem
VO_COMPCHEM_QUEUES="GRIDEGI_large"
VO_COMPCHEM_USERS=vomss://voms.cnaf.infn.it:8443/voms/compchem?/compchem
VO_COMPCHEM_VOMS_SERVERS="'vomss://voms.cnaf.infn.it:8443/voms/compchem?/compchem' 'vomss://voms-01.pd.infn.it:8443/voms/compchem?/compchem'"
VO_COMPCHEM_VOMSES="'compchem voms.cnaf.infn.it 15003 /C=IT/O=INFN/OU=Host/L=CNAF/CN=voms.cnaf.infn.it compchem' 'compchem voms-01.pd.infn.it 15003 /C=IT/O=INFN/OU=Host/L=Padova/CN=voms-01.pd.infn.it compchem'"
VO_COMPCHEM_VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN CA' '/C=IT/O=INFN/CN=INFN CA'"

#::::::::::::::
#dteam
#::::::::::::::
VO_DTEAM_SW_DIR=$VO_SW_DIR/dteam
VO_DTEAM_DEFAULT_SE=$DPM_HOST
VO_DTEAM_STORAGE_DIR=$CLASSIC_STORAGE_DIR/dteam
VO_DTEAM_QUEUES="GRIDEGI_small"
VO_DTEAM_VOMS_SERVERS='vomss://voms.hellasgrid.gr:8443/voms/dteam?/dteam/'
VO_DTEAM_VOMSES="'dteam lcg-voms.cern.ch 15004 /DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch dteam 24' 'dteam voms.cern.ch 15004 /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch dteam 24' 'dteam voms.hellasgrid.gr 1
5004 /C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr dteam 24' 'dteam voms2.hellasgrid.gr 15004 /C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr dteam 24'"
VO_DTEAM_VOMS_CA_DN="'/DC=ch/DC=cern/CN=CERN Trusted Certification Authority' '/DC=ch/DC=cern/CN=CERN Trusted Certification Authority' '/C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2006' '/C=G
R/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2006'"


#::::::::::::::
#fusion
#::::::::::::::
VO_FUSION_SW_DIR=$VO_SW_DIR/fusion
VO_FUSION_DEFAULT_SE=$DPM_HOST
VO_FUSION_STORAGE_DIR=$CLASSIC_STORAGE_DIR/fusion
VO_FUSION_QUEUES="GRIDEGI_large"
VO_FUSION_VOMS_SERVERS="vomss://voms-prg.bifi.unizar.es:8443/voms/fusion?/fusion/"
VO_FUSION_VOMSES="'fusion voms-prg.bifi.unizar.es 15001 /DC=es/DC=irisgrid/O=bifi-unizar/CN=voms-prg.bifi.unizar.es fusion'"
VO_FUSION_VOMS_CA_DN="'/DC=es/DC=irisgrid/CN=IRISGridCA'"


#::::::::::::::
#imath.cesga.es
#::::::::::::::
VO_IMATH_CESGA_ES_SW_DIR=$VO_SW_DIR/imath
VO_IMATH_CESGA_ES_DEFAULT_SE=$DPM_HOST
VO_IMATH_CESGA_ES_STORAGE_DIR=$CLASSIC_STORAGE_DIR/imath
VO_IMATH_CESGA_ES_QUEUES="GRIDEGI_large"
VO_IMATH_CESGA_ES_USERS=vomss://voms.egi.cesga.es:8443/voms/imath.cesga.es?/imath.cesga.es
VO_IMATH_CESGA_ES_VOMS_SERVERS="'vomss://voms.egi.cesga.es:8443/voms/imath.cesga.es?/imath.cesga.es'"
VO_IMATH_CESGA_ES_VOMSES="'imath.cesga.es voms.egi.cesga.es 15100 /DC=es/DC=irisgrid/O=cesga/CN=host/voms.egi.cesga.es imath.cesga.es'"
VO_IMATH_CESGA_ES_VOMS_CA_DN="'/DC=es/DC=irisgrid/CN=IRISGridCA'"


#::::::::::::::
#ops
#::::::::::::::
VO_OPS_SW_DIR=$VO_SW_DIR/ops
VO_OPS_DEFAULT_SE=$DPM_HOST
VO_OPS_STORAGE_DIR=$CLASSIC_STORAGE_DIR/ops
VO_OPS_QUEUES="GRIDEGI_ops"
VO_OPS_VOMS_SERVERS="'vomss://voms.cern.ch:8443/voms/ops?/ops/'"
VO_OPS_VOMSES="'ops voms.cern.ch 15009 /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch ops' 'ops lcg-voms.cern.ch 15009 /DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch ops'"
VO_OPS_VOMS_CA_DN="'/DC=ch/DC=cern/CN=CERN Trusted Certification Authority' '/DC=ch/DC=cern/CN=CERN Trusted Certification Authority'"

## IBERGRID VOS ##
VO_LIFE_VO_IBERGRID_EU_VOMSES="'life.vo.ibergrid.eu voms01.ncg.ingrid.pt 40010 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt life.vo.ibergrid.eu' 'life.vo.ibergrid.eu ibergrid-voms.ifca.es 40010 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es life.vo.ibergrid.eu'"
VO_LIFE_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# hpc.vo.ibergrid.eu
VO_HPC_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/hpcibeusgm
VO_HPC_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_HPC_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/hpcibeu
VO_HPC_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_HPC_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/hpc.vo.ibergrid.eu?/hpc.vo.ibergrid.eu"
VO_HPC_VO_IBERGRID_EU_VOMSES="'hpc.vo.ibergrid.eu voms01.ncg.ingrid.pt 40005 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt hpc.vo.ibergrid.eu' 'hpc.vo.ibergrid.eu ibergrid-voms.ifca.es 40005 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es hpc.vo.ibergrid.eu'"
VO_HPC_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# earth.vo.ibergrid.eu
VO_EARTH_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/earibeusgm
VO_EARTH_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_EARTH_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/earibeu
VO_EARTH_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_EARTH_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/earth.vo.ibergrid.eu?/earth.vo.ibergrid.eu"
VO_EARTH_VO_IBERGRID_EU_VOMSES="'earth.vo.ibergrid.eu voms01.ncg.ingrid.pt 40011 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt earth.vo.ibergrid.eu' 'earth.vo.ibergrid.eu ibergrid-voms.ifca.es 40011 /DC=e
s/DC=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es earth.vo.ibergrid.eu'"
VO_EARTH_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# bing.vo.ibergrid.eu
VO_BING_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/binibeusgm
VO_BING_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_BING_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/binibeu
VO_BING_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_BING_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/bing.vo.ibergrid.eu?/bing.vo.ibergrid.eu"
VO_BING_VO_IBERGRID_EU_VOMSES="'bing.vo.ibergrid.eu voms01.ncg.ingrid.pt 40004 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt bing.vo.ibergrid.eu' 'bing.vo.ibergrid.eu ibergrid-voms.ifca.es 40004 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es bing.vo.ibergrid.eu'"
VO_BING_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# pfound.vo.ibergrid.eu
VO_PFOUND_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/pfoibeusgm
VO_PFOUND_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_PFOUND_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/pfoibeu
VO_PFOUND_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_PFOUND_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/pfound.vo.ibergrid.eu?/pfound.vo.ibergrid.eu"
VO_PFOUND_VO_IBERGRID_EU_VOMSES="'pfound.vo.ibergrid.eu voms01.ncg.ingrid.pt 40006 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt pfound.vo.ibergrid.eu' 'pfound.vo.ibergrid.eu ibergrid-voms.ifca.es 40006 /
DC=es/DC=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es pfound.vo.ibergrid.eu'"
VO_PFOUND_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# phys.vo.ibergrid.eu
VO_PHYS_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/phyibeusgm
VO_PHYS_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_PHYS_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/phyibeu
VO_PHYS_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_PHYS_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/phys.vo.ibergrid.eu?/phys.vo.ibergrid.eu"
VO_PHYS_VO_IBERGRID_EU_VOMSES="'phys.vo.ibergrid.eu voms01.ncg.ingrid.pt 40007 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt phys.vo.ibergrid.eu' 'phys.vo.ibergrid.eu ibergrid-voms.ifca.es 40007 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es phys.vo.ibergrid.eu'"
VO_PHYS_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# social.vo.ibergrid.eu
VO_SOCIAL_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/socibeusgm
VO_SOCIAL_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST

# ops.vo.ibergrid.eu
VO_OPS_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/opsibeusgm
VO_OPS_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_OPS_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/opsibeu
VO_OPS_VO_IBERGRID_EU_QUEUES="GRIDEGI_small"
VO_OPS_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/ops.vo.ibergrid.eu?/ops.vo.ibergrid.eu'"
VO_OPS_VO_IBERGRID_EU_VOMSES="'ops.vo.ibergrid.eu voms01.ncg.ingrid.pt 40001 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt ops.vo.ibergrid.eu' 'ops.vo.ibergrid.eu ibergrid-voms.ifca.es 40001 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es ops.vo.ibergrid.eu'"
VO_OPS_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# tut.vo.ibergrid.eu
VO_TUT_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/tutibeusgm
VO_TUT_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_TUT_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/tutibeu
VO_TUT_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_TUT_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/tut.vo.ibergrid.eu?/tut.vo.ibergrid.eu'"
VO_TUT_VO_IBERGRID_EU_VOMSES="'tut.vo.ibergrid.eu voms01.ncg.ingrid.pt 40002 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt tut.vo.ibergrid.eu' 'tut.vo.ibergrid.eu ibergrid-voms.ifca.es 40002 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es tut.vo.ibergrid.eu'"
VO_TUT_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# iber.vo.ibergrid.eu
VO_IBER_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/iberibeusgm
VO_IBER_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_IBER_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/iberibeu
VO_IBER_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_IBER_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/iber.vo.ibergrid.eu?/iber.vo.ibergrid.eu'"
VO_IBER_VO_IBERGRID_EU_VOMSES="'iber.vo.ibergrid.eu voms01.ncg.ingrid.pt 40003 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt iber.vo.ibergrid.eu' 'iber.vo.ibergrid.eu ibergrid-voms.ifca.es 40003 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es iber.vo.ibergrid.eu'"
VO_IBER_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# eng.vo.ibergrid.eu
VO_ENG_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/engibeusgm 
VO_ENG_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_ENG_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/engibeu 
VO_ENG_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_ENG_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/eng.vo.ibergrid.eu?/eng.vo.ibergrid.eu"
VO_ENG_VO_IBERGRID_EU_VOMSES="'eng.vo.ibergrid.eu voms01.ncg.ingrid.pt 40013 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt eng.vo.ibergrid.eu' 'eng.vo.ibergrid.eu ibergrid-voms.ifca.es 40013 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es eng.vo.ibergrid.eu'"
VO_ENG_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# ict.vo.ibergrid.eu
VO_ICT_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/ictibeusgm 
VO_ICT_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_ICT_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/ictibeu 
VO_ICT_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_ICT_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/ict.vo.ibergrid.eu?/ict.vo.ibergrid.eu"
VO_ICT_VO_IBERGRID_EU_VOMSES="'ict.vo.ibergrid.eu voms01.ncg.ingrid.pt 40008 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt ict.vo.ibergrid.eu' 'ict.vo.ibergrid.eu ibergrid-voms.ifca.es 40008 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es ict.vo.ibergrid.eu'"
VO_ICT_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# life.vo.ibergrid.eu
VO_LIFE_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/lifibeusgm 
VO_LIFE_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_LIFE_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/lifibeu 
VO_LIFE_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_LIFE_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/life.vo.ibergrid.eu?/life.vo.ibergrid.eu"
VO_SOCIAL_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/socibeu
VO_SOCIAL_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_SOCIAL_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/social.vo.ibergrid.eu"?/social.vo.ibergrid.eu""
VO_SOCIAL_VO_IBERGRID_EU_VOMSES="'social.vo.ibergrid.eu voms01.ncg.ingrid.pt 40012 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt social.vo.ibergrid.eu' 'social.vo.ibergrid.eu ibergrid-voms.ifca.es 40012 /
DC=es/DC=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es social.vo.ibergrid.eu'"
VO_SOCIAL_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"

# chem.vo.ibergrid.eu
VO_CHEM_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/cheibeusgm
VO_CHEM_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_CHEM_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/cheibeu
VO_CHEM_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_CHEM_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/chem.vo.ibergrid.eu"?/chem.vo.ibergrid.eu""
VO_CHEM_VO_IBERGRID_EU_VOMSES="'chem.vo.ibergrid.eu voms01.ncg.ingrid.pt 40009 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt chem.vo.ibergrid.eu' 'chem.vo.ibergrid.eu ibergrid-voms.ifca.es 40009 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es chem.vo.ibergrid.eu'"
VO_CHEM_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
  • Groups.conf file used

"/VO=cesga/GROUP=/cesga/ROLE=VO-Admin":::sgm:
"/VO=cesga/GROUP=/cesga"::::
"/VO=compchem/GROUP=/compchem/ROLE=SoftwareManager":::sgm:
"/VO=compchem/GROUP=/compchem"::::
"/VO=fusion/GROUP=/fusion/ROLE=VO-admin":::sgm:
"/VO=fusion/GROUP=/fusion/ROLE=swadmin":::sgm:
"/VO=fusion/GROUP=/fusion/ROLE=production":::prd:
"/VO=fusion/GROUP=/fusion":::: 
"/VO=imath.cesga.es/GROUP=/imath.cesga.es/ROLE=VO-Admin":::sgm:
"/VO=imath.cesga.es/GROUP=/imath.cesga.es"::::
"/dteam/ROLE=lcgadmin":::sgm:
"/dteam/ROLE=production":::prd:
"/dteam"::::
"/auger/ROLE=SoftwareManager":::sgm:
"/auger/ROLE=Production":::prd:
"/auger"::::
"/auger/FNAL"::::
"/mpi/ROLE=SoftwareManager":::sgm:
"/mpi/ROLE=Production":::prd:
"/mpi"::::
"/biomed/ROLE=lcgadmin":::sgm:
"/biomed/ROLE=production":::prd:
"/biomed"::::
"/ops/ROLE=lcgadmin":::sgm:
"/ops"::::
"/VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu"::::
"/VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu"::::
"/VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu"::::
"/VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu"::::
"/VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu"::::
"/VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu":::
"/VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu"::::
"/VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu"::::
"/VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu"::::
"/VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu"::::
"/VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu"::::
"/VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu"::::
"/VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=VO-Admin":::sgm:
"/VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=SW-Admin":::sgm:
"/VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=Production":::prd:
"/VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu"::::
  • (!) Before configuring with YAIM, we decide to share the Users HOMES between the CreamCE and the WNS

Sharing the GRID HOMES mounted front the EMC storage server with the WNs

[root@cream ~]# cat /etc/fstab 
#
# /etc/fstab
# Created by anaconda on Mon Mar 14 16:30:46 2011
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=afb1e256-6b16-493b-9347-48737b03a990 /                       ext3    defaults        1 1
/dev/sda                swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
# GRID HOMES (Shared between the WNs and the new CreamCE(EGI))
193.144.35.13:/Grid_se/home_grid   /mnt/EMC/Grid_se-home_grid    nfs   rw,rsize=8192,wsize=8192,intr,noatime,vers=3,tcp       0 0


# OPT_CESGA_SVG (in order to read the accounting file from SVG in real time)
193.144.35.13:/Optcesga_SVG    /mnt/EMC/Optcesga_SVG    nfs   rw,rsize=8192,wsize=8192,intr,noatime,vers=3,tcp       0 0

(!) We had to set up NFS version 3, because we got problems with NFS version 4, if we tried to execute a chown of a file we got "invalid argument"

  • Creating the necessary files before mounting the directories

mkdir /mnt/EMC/
mkdir /mnt/EMC/Grid_se-home_grid
ln -sf /mnt/EMC/Grid_se-home_grid /home_grid
  • Install necessary packages to mount NFS

yum install nfs-utils.x86_64
yum install rpc2
  • Changing where useradd will create the home for a User

[root@cream home_grid]# cp -aprx /etc/default/useradd /etc/default/useradd.orig

[root@cream home_grid]# cat /etc/default/useradd
# useradd defaults file
GROUP=100
HOME=/home_grid
INACTIVE=-1
EXPIRE=
SHELL=/bin/bash
SKEL=/etc/skel
CREATE_MAIL_SPOOL=yes

Configuring with YAIM (OK)

 /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n MPI_CE -n creamCE -n SGE_utils

[ .... ]

Configuring IPTABLES on CreamCE

#CREAM
-A RH-Firewall-1-INPUT -p tcp --dport 8443 -j ACCEPT

#GSIFTP
-A RH-Firewall-1-INPUT -p tcp --dport 2811 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport 9002 -j ACCEPT

#BDII (EL SITE BDII ESTA EN EL MON)
-A RH-Firewall-1-INPUT -p tcp --dport 2170 -j ACCEPT

Creating the SGE enviornment on the CreamCE
  • Creating a file by hand in /etc/profile.d to load SGE environment on the CreamCE

[root@cream profile.d]# ll | grep -i sge
-rw-r--r-- 1 root root  152 Feb 17 09:56 sge.sh


[root@cream profile.d]# cat /etc/profile.d/sge.sh 
# Define SGE_ROOT directory and SGE commands
export SGE_ROOT=/usr/local/sge/pro
export SGE_CELL=default
. /usr/local/sge/pro/default/common/settings.sh

Mounting the accounting file from SVG

(!) In order to see the accounting file in real time, we have mounted the /opt/cesga from SVG as Read-only file system, and we have created a symbolic link in the local SGE installation in the CreamCE to the accounting file from SVG. This is necessary on the CreamCE service to have information about job status in real time (executed, cancelled, suspended, finished, etc.)

[root@cream bakcup_cream]# cat /etc/fstab 
[ ...... ]
# OPT_CESGA_SVG (in order to read the accounting file from SVG in real time)
193.144.35.13:/Optcesga_SVG    /mnt/EMC/Optcesga_SVG    nfs   rw,rsize=8192,wsize=8192,intr,noatime,nfsvers=3,tcp       0 0

[root@cream home_grid]# mkdir /mnt/EMC/Optcesga_SVG
[root@cream common]# ln -s /mnt/EMC/Optcesga_SVG/opt/cesga/sge/default/common/accounting /usr/local/sge/pro/default/common/accounting

Configuring the queues and the File transfers

##The CREAM sandbox directory name (default /var/cream_sandbox) can be changed using the yaim variable CREAM_SANDBOX_DIR". 
CREAM_SANDBOX_DIR="/home_grid/var/cream_sandbox"
  • After the YAIM configuration the following file is changed:

[root@cream ~]# grep -i sandbox_dir /etc/glite-ce-cream/cream-config.xml
      <parameter name="CREAM_SANDBOX_DIR" value="/home_grid/var/cream_sandbox" />
      <parameter name="CREAM_SANDBOX_DIR" value="/home_grid/var/cream_sandbox" />
  • We needed to create three new queues:

GRIDEGI_large
GRIDEGI_ops
GRIDEGI_small
  • And in order to configure the file transfers, we needed to configure a prolog/epilog on all these queues:

prolog                root@/opt/exp_soft2/SGE_prolog/prolog.sh
epilog                /opt/exp_soft2/SGE_prolog/sge_filestaging --stageout
  • Where /opt/exp_soft2/SGE_prolog/prolog.sh contents:

[root@compute-5-1 exp_soft2]# cat /opt/exp_soft2/SGE_prolog/prolog.sh
#!/bin/bash
chown $USER $JOB_SCRIPT
chgrp `id $USER -gn` $JOB_SCRIPT
chmod 700 $JOB_SCRIPT
su -m -c "/opt/exp_soft2/SGE_prolog/sge_filestaging --stagein" $USER
exit 0 
  • And /opt/exp_soft2/SGE_prolog/sge_filestaging contents:

[root@compute-5-1 exp_soft2]# cat /opt/exp_soft2/SGE_prolog/sge_filestaging 
#! /usr/bin/perl

use Getopt::Long;

my ( $STAGEIN, $FILELIST );

GetOptions(
    'stagein'  => sub { $STAGEIN = 1; $FILELIST = 'SGE_stagein'; },
    'stageout' => sub { $STAGEIN = 0; $FILELIST = 'SGE_stageout'; },
    );

exit 0 unless $FILELIST;

my ( @FILES ) = split( /\@\@\@/, $ENV{$FILELIST} );

foreach ( @FILES ) {
    next unless /^([^\@]+)\@([^\:]+)\:(.+)$/;

    my $localfile     = $1;
    my $remotemachine = $2;
    my $remotefile    = $3;

    if ( $STAGEIN ) {
       system( 'cp', $remotefile, $localfile );
    } else {
       system( 'cp', $localfile, $remotefile );
    }
}

exit( 0 );

(!) The directory /opt/exp_soft2/SGE_prolog was created by hand in order to this directory is being seen in all the WNs

Assigning complex to the jobs and modifying the sge_submit.sh script for it
  • We added the following lines to the /usr/libexec/sge_submit.sh script:

. /etc/lrms/ge_globals.conf #(added on line 37)

local_submit_attributes_file=${GLITE_LOCATION:-/opt/glite}/libexec/sge_local_submit_attributes.sh #(added on line 74)

# Write GE directives according to /etc/lrms/ge_local_submit_wrapper.conf #(added on line 102)
ge_retcode=`$GE_REQS_SCRIPT --user-attrs=${bls_opt_req_file} $bls_opt_queue $bls_tmp_file 2>&1` #(added on line 103)
[ '$ge_retcode' != '0' ] && echo "$ge_retcode" 1>&2 #(added on line 104)
  • Where:
    • /etc/lrms/ge_globals.conf file is used to load the /usr/libexec/ge_local_submit_wrapper script to parser the complex from the /etc/lrms/ge_local_submit_wrapper.conf file (in which, we configure the complex that we want to assign to the jobs). Its content is:

[root@cream ~]# cat /etc/lrms/ge_globals.conf 
GE_SHARED_SANDBOX=yes
GE_REQS_SCRIPT=/usr/libexec/ge_local_submit_wrapper
  • It is necessary modify the /usr/libexec/ge_local_submit_wrapper script to configure /etc/lrms/ge_local_submit_wrapper.conf file. Modify the line 10 to add the following content to the variable filename:

filename = '/etc/lrms/ge_local_submit_wrapper.conf'
  • Configuring the respective complex for the submitted jobs, these complex are added to the job when it is submitted:

[root@cream ~]# cat /etc/lrms/ge_local_submit_wrapper.conf 
*|complex_values|num_proc=1,h_stack=32M,arch=x86_64,processor=opteron_6174,h_fsize=10G
*|s_rt|150:00:00
*|s_vmem|1G
GRIDEGI_ops|s_vmem|4G

(!) The '*' means that the complex are applied to all the jobs, but it is possible to substitute the '*' for the name of the queue to which we want to apply the complex. (!) WARNING: The "-cwd" option must not be NEVER used, because otherwise it takes the /var/tmp directory as HOME because the qsub is executed from the /var/tmp directory, and "-cwd" option what it does is to use current working directory.

  • This is the content for /usr/libexec/sge_submit.sh script once modified:

[root@cream ~]# cat /usr/libexec/sge_submit.sh
#!/bin/bash
#
# File:     sge_submit.sh
# Author:   Keith Sephton (kms@doc.ic.ac.uk)
#
# Based on pbs_submit.sh 
# Author:   David Rebatto (david.rebatto@mi.infn.it)
#
# Revision history:
#    xx-Apr-2008: Original release
#    11-Nov-2009: Mario David (david@lip.pt). Removed CELL information from $jobID
#
# Description:
#   Submission script for SGE, to be invoked by blahpd server.
#   Usage:
#     sge_submit.sh -c <command> [-i <stdin>] [-o <stdout>] [-e <stderr>] [-w working dir] [-- command's arguments]
#
# Copyright (c) Members of the EGEE Collaboration. 2004. 
# See http://www.eu-egee.org/partners/ for details on the copyright
# holders.  
# 
# Licensed under the Apache License, Version 2.0 (the "License"); 
# you may not use this file except in compliance with the License. 
# You may obtain a copy of the License at 
# 
#     http://www.apache.org/licenses/LICENSE-2.0 
# 
# Unless required by applicable law or agreed to in writing, software 
# distributed under the License is distributed on an "AS IS" BASIS, 
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 
# See the License for the specific language governing permissions and 
# limitations under the License.
#

#exec 2>> /tmp/submit.log

. /etc/lrms/ge_globals.conf

. `dirname $0`/blah_common_submit_functions.sh

if [ -z "$sge_rootpath" ]; then sge_rootpath="/usr/local/sge/pro"; fi
if [ -r "$sge_rootpath/${sge_cellname:-default}/common/settings.sh" ]
then
  . $sge_rootpath/${sge_cellname:-default}/common/settings.sh
fi

bls_job_id_for_renewal=JOB_ID

original_args="$@"
bls_parse_submit_options "$@"

bls_setup_all_files

# Write wrapper preamble
cat > $bls_tmp_file << end_of_preamble
#!/bin/bash
# SGE job wrapper generated by `basename $0`
# on `/bin/date`

module load grid/egi

module load mpich2/1.3.2p1-gnu

#
# stgcmd = $bls_opt_stgcmd
# proxy_string = $bls_opt_proxy_string
# proxy_local_file = $bls_proxy_local_file
#
# SGE directives:
#\$ -S /bin/bash
end_of_preamble

#local batch system-specific file output must be added to the submit file
local_submit_attributes_file=${GLITE_LOCATION:-/opt/glite}/libexec/sge_local_submit_attributes.sh
if [ -r $local_submit_attributes_file ] ; then
    echo \#\!/bin/sh > $bls_opt_tmp_req_file
    if [ ! -z $bls_opt_req_file ] ; then
        cat $bls_opt_req_file >> $bls_opt_tmp_req_file
    fi
    echo "source $local_submit_attributes_file" >> $bls_opt_tmp_req_file
    chmod +x $bls_opt_tmp_req_file
    $bls_opt_tmp_req_file >> $bls_tmp_file 2> /dev/null
    rm -f $bls_opt_tmp_req_file
fi

if [ ! -z "$bls_opt_xtra_args" ] ; then
    echo -e $bls_opt_xtra_args >> $bls_tmp_file 2> /dev/null
fi

# Write SGE directives according to command line options
# handle queue overriding
[ -z "$bls_opt_queue" ] || grep -q "^#\$ -q" $bls_tmp_file || echo "#\$ -q $bls_opt_queue" >> $bls_tmp_file
[ -z "$bls_opt_mpinodes" -o "x${bls_opt_mpinodes}" = "x1" ] || grep -q "^#\$ -pe *\\*" $bls_tmp_file || echo "#\$ -pe * $bls_opt_mpinodes" >>$bls_tmp_file

# Input and output sandbox setup.
bls_fl_subst_and_accumulate inputsand "@@F_REMOTE@`hostname -f`:@@F_LOCAL" "@@@"
[ -z "$bls_fl_subst_and_accumulate_result" ] || echo "#\$ -v SGE_stagein=$bls_fl_subst_and_accumulate_result" >> $bls_tmp_file
bls_fl_subst_and_accumulate outputsand "@@F_REMOTE@`hostname -f`:@@F_LOCAL" "@@@"
[ -z "$bls_fl_subst_and_accumulate_result" ] || echo "#\$ -v SGE_stageout=$bls_fl_subst_and_accumulate_result" >> $bls_tmp_file
echo "#$ -m n"  >> $bls_tmp_file

# Write GE directives according to /etc/lrms/ge_local_submit_wrapper.conf
ge_retcode=`$GE_REQS_SCRIPT --user-attrs=${bls_opt_req_file} $bls_opt_queue $bls_tmp_file 2>&1`
[ '$ge_retcode' != '0' ] && echo "$ge_retcode" 1>&2

bls_add_job_wrapper

###############################################################
# Submit the script
###############################################################
#Your job 3236842 ("run") has been submitted
jobID=`qsub $bls_tmp_file 2> /dev/null | perl -ne 'print $1 if /^Your job (\d+) /;'` # actual submission

#jobID=`qsub -l s_rt=540000,s_vmem=4G,num_proc=1,h_stack=32M,arch=x86_64,processor=opteron_6174,h_fsize=10 $bls_tmp_file 2> /dev/null | perl -ne 'print $1 if /^Your job (\d+) /;'` # actual submission

retcode=$?
if [ "$retcode" != "0" -o -z "$jobID" ] ; then
        rm -f $bls_tmp_file
        exit 1
fi
# 11/11/09 Mario David fix (remove CELL)
#jobID=$jobID.${SGE_CELL:-default}

# Compose the blahp jobID ("sge/" + datenow + sge jobid)
# 11/11/09 Mario David fix 
blahp_jobID=sge/`date +%Y%m%d%H%M%S`/$jobID

if [ "x$job_registry" != "x" ]; then
  now=`date +%s`
  let now=$now-1
  ${blah_sbin_directory}/blah_job_registry_add "$blahp_jobID" "$jobID" 1 $now "$bls_opt_creamjobid" "$bls_proxy_local_file" "$bls_opt_proxyrenew_numeric" "$bls_opt_proxy_subject"
fi

echo "BLAHP_JOBID_PREFIX$blahp_jobID"
bls_wrap_up_submit

exit $retcode

(!) The module lines at the beginning are to load the Grid Engine and MPI environment on the Worker Nodes.

Configuring MPI
  • Creating a GE parallel environment

[root@cream ldif]# qconf -ap GRIDEGI_mpi
root@cream.egi.cesga.es added "GRIDEGI_mpi" to parallel environment list

[root@cream ldif]# qconf -mp GRIDEGI_mpi
pe_name            GRIDEGI_mpi
slots              9999
user_lists         NONE
xuser_lists        NONE
start_proc_args    /opt/cesga/sistemas/sge/tools/util/startmpi.sh -catch_rsh \
                   $pe_hostfile
stop_proc_args     /opt/cesga/sistemas/sge/tools/util/stopmpi.sh
allocation_rule    $pe_slots
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary FALSE
  • Configuring the parallel environment in the GRID queues:

[root@cream ldif]# qconf -sq GRIDEGI_large | grep -i pe_
pe_list               GRIDEGI_mpi

[root@cream ldif]# qconf -sq GRIDEGI_small | grep -i pe_
pe_list               GRIDEGI_mpi

[root@cream ldif]# qconf -sq GRIDEGI_ops | grep -i pe_
pe_list               GRIDEGI_mpi
  • We added the following command on line 60 in the /usr/bin/sge_submit.sh script to load the right MPI environment and compilers. This is because we used the mpich installed on SVG cluster and we load the environment on the cluster using module application:

module load mpich2/1.3.2p1-gnu 

BUGs/Known issues found after reconfiguring with YAIM

Problem at first configuration with EMI-2 CREAM with GE

The first yaim configuration for a EMI-2 CREAM CE using GE as batch system fails with:

/etc/lrms/scheduler.conf: No such file or directory 

The problem disappears running again yaim 

Problem if there are "extra" characters in the beginning of the tomcat key file

Because on an issue in trustmanager, there can be problems if there is something before -----BEGIN ...---- in /etc/grid-security/tomcat-key.pem

The workaround is to simply remove these chars 

Bug on the sge_submit.sh script

https://savannah.cern.ch/bugs/?95385
[root@cream grid-security]# diff /usr/libexec/sge_submit.sh /usr/libexec/sge_submit.sh_original
86c86
< [ -z "$bls_opt_mpinodes" -o "x${bls_opt_mpinodes}" = "x1" ] || grep -q "^#\$ -pe *\\*" $bls_tmp_file || echo "#\$ -pe * $bls_opt_mpinodes" >>$bls_tmp_file
---
> [ -z "$bls_opt_mpinodes" -o "x${bls_opt_mpinodes}" = "x1" ] || grep -q"^#\$ -pe *\\*" $bls_tmp_file || echo "#\$ -pe * $bls_opt_mpinodes" >>$bls_tmp_file

Removing sge ports from /etc/services

[root@cream ~]# grep 6444 /etc/services 
sge_qmaster     6444/tcp                # Grid Engine Qmaster Service
sge_qmaster     6444/udp                # Grid Engine Qmaster Service


[root@cream ~]# grep 6444 /etc/services 
###sge_qmaster     6444/tcp                # Grid Engine Qmaster Service
###sge_qmaster     6444/udp                # Grid Engine Qmaster Service

It was necessary to load the SGE environment on the /var/lib/bdii/gip/plugin/glite-info-dynamic-ce script

[root@cream opssgm]# cat /var/lib/bdii/gip/plugin/glite-info-dynamic-ce
#!/bin/sh

. /etc/profile.d/sge.sh

/usr/libexec/glite-ce-check-submission-state /etc/glite-ce-dbtool/creamdb_min_access.conf
retcode=$?
if [ $retcode = 1 ] || [ $retcode = 2 ] ; then
  # Submissions disabled. Publish Draining
  /usr/libexec/glite-info-dynamic-sge --info -c /etc/lrms/scheduler.conf | sed -e 's/GlueCEStateStatus: Production/GlueCEStateStatus: Draining/'
else 
  # Submissions enabled. The default cluster.state will be used
  /usr/libexec/glite-info-dynamic-sge --info -c /etc/lrms/scheduler.conf
fi

Modifying /var/lib/bdii/gip/ldif/ComputingShare.ldif

(!) It was necessary to edit /var/lib/bdii/gip/ldif/ComputingShare.ldif file and substitute the empty GLUE2PolicyRule attribute by GLUE2PolicyRule: VO:<name of the VO> :

Publishing the GlueCEPolicyMaxSlotsPerJob to run MPI jobs correctly

  • Go to:

[root@cream ldif]# pwd
/var/lib/bdii/gip/ldif
  • And modify static-file-CE.ldif file to add on GlueCEPolicyMaxSlotsPerJob variable the maximum number of slots offered for MPI jobs:

[root@cream ldif]#   grep -R GlueCEPolicyMaxSlotsPerJob ../ldif/*
../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8
../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8
../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8
[ ..... ]

(!) If you obtain 999999, please substitute the static-file-CE.ldif by this one /root/static-file-CE.ldif_modificado

Increase the value for the memory in /usr/bin/apel-sge-log-parser

(!) We had to change the following value in /usr/bin/apel-sge-log-parser file, because otherwise the APEL crond was not able to work. In our case, we have a big accounting file, therefore, the APEL crond needs pretty memory to be able to process the accounting file:

if [ -z $JAVA_HOME ]; then
  java -Xmx2524m -classpath $CP $LOG4J_PROPS org.glite.apel.sge.ApelLogParser "$@"
else
  $JAVA_HOME/bin/java -Xmx2524m -classpath $CP $LOG4J_PROPS org.glite.apel.sge.ApelLogParser "$@"
fi
 
exit $?

The BUpdaterSGE daemons does not inherits the correct GE environment variables

(!) Jobs were submitted without requesting complex due to this issue

If the environment present in a BUpdaterSGE process does not include the GE environment variables, the GE client commands (qstat, qconf) can not be executed by BUpdaterSGE.

As a consequence, BUpdaterSGE will assume that jobs have been cancelled (because it receives no information from qstat or qacct). You can check the environment for BUpdaterSGE process using the following commands and searching for the GE env variables (SGE_EXECD, SGE_QMASTER, SGE_ROOT, SGE_CLUSTER_NAME, SGE_CELL)

# ps xuawww | grep -i sge
tomcat    7423  0.6  0.5  37184 21328 ?        S    Nov23 103:56 /usr/bin/BUpdaterSGE
root     30622  0.0  0.0  61180   804 pts/0    R+   13:41   0:00 grep -i sge

# cat /proc/7423/environ 

This can happen if the BUpdaterSGE daemon is restarted by other user different than root (for example, tomcat starts the daemon at boot time and restarts it if the daemon is dead) without sourcing the proper environment. The workaround is to force the environment to be loaded in /etc/init.d/gLite and /etc/init.d/glite-ce-blahparser. This can be done simply by adding a line like the one bellow to be sourced at the beguinning of previous scripts

 . /etc/profile.d/sge.sh

Things to have in mind before reconfiguring with YAIM after some update

  • Do backup of:

/etc/init.d/gLite 
/etc/init.d/glite-ce-blahparser 

(!) On these scripts must be loaded the SGE environment because otherwise jobs are submitted without requesting complex, this can be solved simply by adding a line like the one bellow to be sourced at the beginning of previous scripts

 . /etc/profile.d/sge.sh
  • Make backup of the following files

cd /var/lib/bdii/gip/plugin
[root@cream plugin]# cp -aprx glite-info-dynamic-ce glite-info-dynamic-ce_mod

[root@cream plugin]# ll /usr/libexec/sge_submit.sh*
-rwxr-xr-x 1 root root 4861 Feb 19 11:40 /usr/libexec/sge_submit.sh
-rwxr-xr-x 1 root root 4861 Feb 19 11:40 /usr/libexec/sge_submit.sh_mod_OK

[root@cream libexec]# cp -aprx /var/lib/bdii/gip/ldif/ComputingShare.ldif /var/lib/bdii/gip/ldif/ComputingShare.ldif_mod_OK
[root@cream libexec]# 
  • Publishing the GlueCEPolicyMaxSlotsPerJob to run MPI jobs correctly
    • Go to:

[root@cream ldif]# pwd
/var/lib/bdii/gip/ldif
  • And check the you get the following output:

[root@cream ldif]#   grep -R GlueCEPolicyMaxSlotsPerJob ../ldif/*
../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8
../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8
../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8
[ ..... ]

TESTING

Checking ldap (OK)

[root@ce2 ~]# ldapsearch -x -H ldap://cream.egi.cesga.es:2170 -b o=grid  | grep 4444

[root@ce2 ~]#  ldapsearch -x -H ldap://cream.egi.cesga.es:2170 -b o=grid  | grep GlueCEStateWaitingJobs
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0
GlueCEStateWaitingJobs: 0

[root@ce2 ~]# ldapsearch -x -H ldap://cream.egi.cesga.es:2170 -b o=grid  | grep GlueCEStateFreeCPUs
GlueCEStateFreeCPUs: 90
GlueCEStateFreeCPUs: 27
GlueCEStateFreeCPUs: 58

[root@ce2 ~]# ldapsearch -x -H ldap://cream.egi.cesga.es:2170 -b o=grid  | grep -i mpi
[ .... ]
GlueHostApplicationSoftwareRunTimeEnvironment: MPI-START
GlueHostApplicationSoftwareRunTimeEnvironment: MPI-START-1.2.0
GlueHostApplicationSoftwareRunTimeEnvironment: MPICH2
GlueHostApplicationSoftwareRunTimeEnvironment: MPICH2-1.3.2p1
GlueHostApplicationSoftwareRunTimeEnvironment: MPI_SHARED_HOME

Submitting a job from the CESGA UI

[esfreire@ui ~]$ voms-proxy-init --voms ops:/ops/Role=lcgadmin
Enter GRID pass phrase:
Your identity: /DC=es/DC=irisgrid/O=cesga/CN=jesteban.freire
Creating temporary proxy ............................ Done
Contacting  voms.cern.ch:15009 [/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch] "ops" Done
Creating proxy ................................................................................... Done

Your proxy is valid until Thu Feb 23 22:56:01 2012

[esfreire@ui ~]$  glite-ce-delegate-proxy -e cream.egi.cesga.es esfreire
2012-02-23 11:02:14,272 NOTICE - Proxy with delegation id [esfreire] succesfully delegated to endpoint [https://cream.egi.cesga.es:8443//ce-cream/services/gridsite-delegation]

[esfreire@ui ~]$  glite-ce-proxy-renew -e cream.egi.cesga.es esfreire
2012-02-23 11:02:26,324 NOTICE - Proxy with delegation id [esfreire] succesfully renewed to endpoint [https://cream.egi.cesga.es:8443//ce-cream/services/gridsite-delegation]

[esfreire@ui ~]$ glite-ce-job-submit -D esfreire -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops lanzar_gromacs.jdl 
https://cream.egi.cesga.es:8443/CREAM421374529

[esfreire@ui ~]$  glite-ce-job-status https://cream.egi.cesga.es:8443/CREAM421374529

******  JobID=[https://cream.egi.cesga.es:8443/CREAM421374529]
        Status        = [DONE-OK]
        ExitCode      = [W]

[root@se2 tmp]# ll /tmp/ | grep -i gro
-rw-r--r-- 1 opssgm004   opssgm   221898 Feb 23 11:05 job_gromacs_dir_output.tar.gz
-rw-r--r-- 1 opssgm004   opssgm   103053 Feb 23 11:05 test_gromacs.err
-rw-r--r-- 1 opssgm004   opssgm    44018 Feb 23 11:05 test_gromacs.out
  • Using the gLite-WMS

[esfreire@ui ~]$ glite-wms-job-submit -a -o test_cream -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops lanzar_gromacs.jdl

Connecting to the service https://wms01.egee.cesga.es:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://wms01.egee.cesga.es:9000/yG5zkzKwYNwVUTkJa-RlVg

The job identifier has been saved in the following file:
/home/esfreire/test_cream

==========================================================================

[esfreire@ui ~]$  glite-wms-job-status -i test_cream


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://wms01.egee.cesga.es:9000/yG5zkzKwYNwVUTkJa-RlVg
Current Status:     Done (Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops
Submitted:          Thu Feb 23 16:22:32 2012 CET
==========================================================================

esfreire@ui ~]$  glite-wms-job-output --dir jobOutput/ -i  test_cream

Connecting to the service https://wms01.egee.cesga.es:7443/glite_wms_wmproxy_server


================================================================================

                        JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://wms01.egee.cesga.es:9000/yG5zkzKwYNwVUTkJa-RlVg
have been successfully retrieved and stored in the directory:
/home/esfreire/jobOutput/esfreire_yG5zkzKwYNwVUTkJa-RlVg

================================================================================

Testing MPI jobs

[esfreire@ui mpi_test_CreamCE_prod]$ cat job1.jdl 
CPUNumber     = 4;
Executable    = "/opt/exp_soft2/mpi-start/bin/mpi-start";
Arguments     = "-v -pre hooks.sh cpi";
InputSandbox  = {"cpi.c", "hooks.sh"};
StdOutput     = "std.out";
StdError      = "std.err";
OutputSandbox = {"std.out","std.err"};
##OutputSandboxBaseDestUri = "gsiftp://localhost";
OutputSandboxBaseDestUri = "gsiftp://se2.egi.cesga.es/tmp";
Requirements  =
    Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment) &&
    Member("OPENMPI", other.GlueHostApplicationSoftwareRunTimeEnvironment);

[esfreire@ui mpi_test_CreamCE_prod]$ cat hooks.sh 
#!/bin/sh

pre_run_hook () {

  # Compile the program.
  echo "Compiling ${I2G_MPI_APPLICATION}"

  sleep 20

  # Actually compile the program.
  cmd="mpicc ${MPI_MPICC_OPTS} -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c"
  $cmd
  if [ ! $? -eq 0 ]; then
    echo "Error compiling program.  Exiting..."
    return 1
  fi

  # Everything's OK.
  echo "Successfully compiled ${I2G_MPI_APPLICATION}"

  return 0
}

Applying 4 CPUS (OK)

[esfreire@ui mpi_test_CreamCE_prod]$  glite-ce-job-submit -D esfreire -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops job1.jdl
https://cream.egi.cesga.es:8443/CREAM390858921


[esfreire@ui mpi_test_CreamCE_prod]$ glite-ce-job-status https://cream.egi.cesga.es:8443/CREAM390858921

******  JobID=[https://cream.egi.cesga.es:8443/CREAM390858921]
        Status        = [DONE-OK]
        ExitCode      = [0]

[root@se2 tmp]# cat std.err
mpi-start [INFO   ]: ***********************************************************
mpi-start [INFO   ]:  UID     =  opssgm002
mpi-start [INFO   ]:  HOST    =  compute-5-1.local
mpi-start [INFO   ]:  DATE    =  Tue Feb 28 10:17:48 CET 2012
mpi-start [INFO   ]:  VERSION =  1.3.0
mpi-start [INFO   ]: ***********************************************************
mpi-start [INFO   ]: search for scheduler
mpi-start [INFO   ]: activate support for sge
mpi-start [INFO   ]: Detected 2 CPU socket(s) and 12 core(s) per CPU
mpi-start [INFO   ]: activate support for generic
mpi-start [INFO   ]: call backend MPI implementation
mpi-start [INFO   ]: start program with mpirun
Process 0 on compute-5-1.local: n=0
Using 16384 intervals
Process 1 on compute-5-1.local: n=0
Process 2 on compute-5-1.local: n=0
Process 3 on compute-5-1.local: n=0


[root@se2 tmp]# cat std.out
Compiling cpi
Successfully compiled cpi
=[START]=======================================================================
pi is approximately 3.1415926539002341, Error is 0.0000000003104410
wall clock time = 0.000208
=[FINISHED]====================================================================

Applying 8 CPUS (OK)

[root@se2 tmp]# cat std.err
mpi-start [INFO   ]: ***********************************************************
mpi-start [INFO   ]:  UID     =  opssgm002
mpi-start [INFO   ]:  HOST    =  compute-5-1.local
mpi-start [INFO   ]:  DATE    =  Tue Feb 28 10:29:10 CET 2012
mpi-start [INFO   ]:  VERSION =  1.3.0
mpi-start [INFO   ]: ***********************************************************
mpi-start [INFO   ]: search for scheduler
mpi-start [INFO   ]: activate support for sge
mpi-start [INFO   ]: Detected 2 CPU socket(s) and 12 core(s) per CPU
mpi-start [INFO   ]: activate support for generic
mpi-start [INFO   ]: call backend MPI implementation
mpi-start [INFO   ]: start program with mpirun
Process 0 on compute-5-1.local: n=0
Using 16384 intervals
Process 1 on compute-5-1.local: n=0
Process 2 on compute-5-1.local: n=0
Process 3 on compute-5-1.local: n=0
Process 4 on compute-5-1.local: n=0
Process 5 on compute-5-1.local: n=0
Process 6 on compute-5-1.local: n=0
Process 7 on compute-5-1.local: n=0

[root@se2 tmp]# cat std.out
Compiling cpi
Successfully compiled cpi
=[START]=======================================================================
pi is approximately 3.1415926539002332, Error is 0.0000000003104401
wall clock time = 0.000416
=[FINISHED]====================================================================

Second MPI job - Applying 4 CPUS (OK)

[esfreire@ui mpi_test_CreamCE_prod]$  glite-ce-job-submit -D esfreire -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops job3.jdl
https://cream.egi.cesga.es:8443/CREAM011898581


[root@se2 tmp]# cat std.err
mpi-start [INFO   ]: ***********************************************************
mpi-start [INFO   ]:  UID     =  opssgm002
mpi-start [INFO   ]:  HOST    =  compute-5-1.local
mpi-start [INFO   ]:  DATE    =  Tue Feb 28 10:35:23 CET 2012
mpi-start [INFO   ]:  VERSION =  1.3.0
mpi-start [INFO   ]: ***********************************************************
mpi-start [INFO   ]: search for scheduler
mpi-start [INFO   ]: activate support for sge
mpi-start [INFO   ]: Detected 2 CPU socket(s) and 12 core(s) per CPU
mpi-start [INFO   ]: activate support for generic
mpi-start [INFO   ]: call backend MPI implementation
mpi-start [INFO   ]: start program with mpirun
Process 2 on compute-5-1.local: n=0
Process 0 on compute-5-1.local: n=0
Using 16384 intervals
Process 1 on compute-5-1.local: n=0
Process 3 on compute-5-1.local: n=0

[root@se2 tmp]# cat std.out
Compiling cpi
Successfully compiled cpi
=[START]=======================================================================
pi is approximately 3.1415926539002341, Error is 0.0000000003104410
wall clock time = 0.000544
=[FINISHED]====================================================================

jobOldSyntax - Applying 4 CPUS (OK)

glite-ce-job-submit -D esfreire -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops job-oldsyntax.jdl 


[esfreire@ui mpi_test_CreamCE_prod]$ cat job-oldsyntax.jdl
CPUNumber     = 4;
Executable    = "starter.sh";
Arguments     = "cpi";
InputSandbox  = {"starter.sh", "cpi.c", "hooks.sh"};
StdOutput     = "std.out";
StdError      = "std.err";
OutputSandbox = {"std.out","std.err"};
OutputSandboxBaseDestUri = "gsiftp://se2.egi.cesga.es/tmp";
Environment   = {"I2G_MPI_PRE_RUN_HOOK=hooks.sh"};
Requirements  =
    Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment) &&
    Member("OPENMPI", other.GlueHostApplicationSoftwareRunTimeEnvironment);


[root@se2 tmp]# cat std.err 
mpi-start [INFO   ]: ***********************************************************
mpi-start [INFO   ]:  UID     =  opssgm002
mpi-start [INFO   ]:  HOST    =  compute-5-1.local
mpi-start [INFO   ]:  DATE    =  Tue Feb 28 10:43:11 CET 2012
mpi-start [INFO   ]:  VERSION =  1.3.0
mpi-start [INFO   ]: ***********************************************************
mpi-start [INFO   ]: search for scheduler
mpi-start [INFO   ]: activate support for sge
mpi-start [INFO   ]: Detected 2 CPU socket(s) and 12 core(s) per CPU
mpi-start [INFO   ]: activate support for generic
mpi-start [INFO   ]: call backend MPI implementation
mpi-start [INFO   ]: start program with mpirun
Process 0 on compute-5-1.local: n=0
Using 16384 intervals
Process 1 on compute-5-1.local: n=0
Process 3 on compute-5-1.local: n=0
Process 2 on compute-5-1.local: n=0

[root@se2 tmp]# cat std.out 
Compiling cpi
Successfully compiled cpi
=[START]=======================================================================
pi is approximately 3.1415926539002341, Error is 0.0000000003104410
wall clock time = 0.000533
=[FINISHED]====================================================================