Contents
-
EMI_creamCE_GRID_Engine installation
- Interesting links
- GE installation links
- MPI installation links
- Preparing the installation and repositories
-
Installing
- Installing the CREAM-CE software
- Installation of the batch system specific software
- Configure the GE batch system software before installing and configuring the CREAM software
- Installing MPI packages
- Configuration of a CREAM CE node in no cluster mode
- Configure the siteinfo.def file
- Sharing the GRID HOMES mounted front the EMC storage server with the WNs
-
BUGs/Known issues found after reconfiguring with YAIM
- Problem at first configuration with EMI-2 CREAM with GE
- Problem if there are "extra" characters in the beginning of the tomcat key file
- Bug on the sge_submit.sh script
- Removing sge ports from /etc/services
- It was necessary to load the SGE environment on the /var/lib/bdii/gip/plugin/glite-info-dynamic-ce script
- Modifying /var/lib/bdii/gip/ldif/ComputingShare.ldif
- Publishing the GlueCEPolicyMaxSlotsPerJob to run MPI jobs correctly
- Increase the value for the memory in /usr/bin/apel-sge-log-parser
- The BUpdaterSGE daemons does not inherits the correct GE environment variables
- Things to have in mind before reconfiguring with YAIM after some update
- TESTING
EMI_creamCE_GRID_Engine installation
Interesting links
- htps://twiki.cern.ch/twiki/bin/view/EMI/EMI2Final#gLite_MPI_v_1_3_0_task_23330
https://wiki.italiangrid.it/twiki/bin/view/CREAM/TroubleshootingGuide
https://wiki.italiangrid.it/twiki/bin/view/CREAM/KnownIssues
GE installation links
MPI installation links
https://twiki.cern.ch/twiki/bin/view/EMI/EMI2Final#gLite_MPI_v_1_3_0_task_23330
https://grid.ifca.es/wiki/Middleware/MpiStart/SiteConfiguration
https://grid.ifca.es/wiki/Middleware/MpiStart/TroubleshootingGuide
Preparing the installation and repositories
- Install yum-priorities RPM and protectbase
yum install yum-priorities yum install yum-protectbase.noarch
- Disabling yum auto-update
wget http://forge.cnaf.infn.it/frs/download.php/101/disable_yum.sh sh -x disable_yum.sh
Installing
- Installing the CAs
yum install ca-policy-egi-core yum install lcg-CA
Installing the CREAM-CE software
- Install the CREAM-CE metapackage:
- yum install emi-cream-ce
==================================================================================================================================================================================================================== Package Arch Version Repository Size ==================================================================================================================================================================================================================== Installing: emi-cream-ce x86_64 1.1.0-4.sl6 UMD-2-updates 2.5 k [ .... ]
Installation of the batch system specific software
READ the next advertisement if you are installing a CreamCe with GE
You have to install and configure the GE batch system software before installing and configuring the CREAM software. The CREAM CE integration was tested with GE 6.2u5 but it should work with any forked version of the original GE software. The support of the GE batch system software (or any of its forked versions) is out of the scope of this activity.
Before proceeding, please take note of the following remarks:
CREAM CE must be installed in a separate node from the GE SERVER (GE QMASTER).
CREAM CE must work as a GE submission host (use qconf -as <CE.MY.DOMAIN> in the GE QMASTER to set it up). - After the installation of the CREAM CE metapackage it is necessary to install the batch system specific metapackage(s):
- If you are running Torque, and your CREAM CE node is the torque master, install the emi-torque-server and emi-torque-utils metapackages:
- If you are running GE, install the emi-ge-utils metapackage:
yum install emi-ge-utils [ ... ] ==================================================================================================================================================================================================================== Package Arch Version Repository Size ==================================================================================================================================================================================================================== Installing: emi-ge-utils x86_64 2.0.0-1.sl6 UMD-2-updates 1.9 k Installing for dependencies: glite-apel-sge noarch 2.0.8-8.sl6 UMD-2-updates 24 k glite-info-dynamic-ge noarch 6.0.0-4.sl6 UMD-2-updates 12 k glite-yaim-ge-utils noarch 4.3.0-4.sl6 UMD-2-updates 17 k perl-XML-LibXML x86_64 1:1.70-5.el6 sl 363 k perl-XML-NamespaceSupport noarch 1.10-3.el6 sl 16 k perl-XML-SAX noarch 0.96-7.el6 sl 77 k perl-XML-Twig noarch 3.34-1.el6 sl 200 k Transaction Summary ====================================================================================================================================================================================================================
Configure the GE batch system software before installing and configuring the CREAM software
- In our case, we are using the GE batch system installed on SVG cluster, therefore, we need to use the same GE version in the CreamCE that in the SVG, so we have copied the GE installation directory from SVG to the CreamCE, and we have done the following steps in order to be able to execute GE commands from the CreamCE and be able to submit jobs to it:
- Copying the GE directory
cd /usr/local tar cvzf sge.tar.gz sge scp -r sge.tar.gz root@cream.egi.cesga.es:/usr/local/
- And on CreamCE side:
cd /usr/local;tar -zvxf sge.tar.gz
- Giving administrative and submission permissions to the new cream.egi.cesga.es on the system where is installed/configured the GE batch system.
[root@cream ~]# qconf -help | grep as [-as hostname_list] add a submit host [root@cream ~]# qconf -help | grep ah [-ah hostname_list] add an administrative host [-ahgrp group] add new host group entry
qconf -as cream.egi.cesga.es qconf -ah cream.egi.cesga.es
- Checking that the changes were done correctly:
[root@ce2 local]# qconf -ss | grep -i cream cream.egi.cesga.es [root@ce2 local]# qconf -sh | grep -i cream cream.egi.cesga.es
- Putting the new CreamCE in the /etc/hosts in the SVG in order to resolve to it:
[root@svgd ~]# grep -i cream /etc/hosts 84.21.169.124 cream.egi.cesga.es
Installing MPI packages
Installing the MPI software on CreamCe (Install the mpi metapackage):
- Useful links:
- yum install mpi-start
==================================================================================================================================================================================================================== Package Arch Version Repository Size ==================================================================================================================================================================================================================== Installing: mpi-start noarch 1.3.0-1.el6 UMD-2-updates 38 k Transaction Summary ====================================================================================================================================================================================================================
- yum install emi-mpi
==================================================================================================================================================================================================================== Package Arch Version Repository Size ==================================================================================================================================================================================================================== Installing: emi-mpi noarch 1.0.2-1.el6 UMD-2-updates 1.9 k Installing for dependencies: glite-yaim-mpi noarch 1.1.11-1.el6 UMD-2-updates 14 k Transaction Summary ====================================================================================================================================================================================================================
Configuration of a CREAM CE node in no cluster mode
- Make sure to place the two files in the target node into the /etc/grid-security directory. Then set the proper mode and ownerships doing:
chown root.root /etc/grid-security/hostcert.pem chown root.root /etc/grid-security/hostkey.pem chmod 600 /etc/grid-security/hostcert.pem chmod 400 /etc/grid-security/hostkey.pem
Configure the siteinfo.def file
- Set your siteinfo.def file, which is the input file used by yaim. Documentation about yaim variables relevant for CREAM CE is available at:
https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#cream_CE
https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#SGE_GE_utils
- Site-info.def file used
# YAIM example site configuration file - adapt it to your site!
MY_DOMAIN=egee.cesga.es
# Node names
# Note: - SE_HOST --> Removed, see CLASSIC_HOST, DCACHE_ADMIN, DPM_HOST below
# - REG_HOST --> There is only 1 central registry for the time being.
CE_HOST=cream.egi.cesga.es
LB_HOST=wms01.egi.cesga.es
WMS_HOST=wms01.egi.cesga.es
PX_HOST=myproxy.egi.cesga.es
BDII_HOST=topbdii.core.ibergrid.eu
BDII_LIST="topbdii.core.ibergrid.eu:2170"
MON_HOST=apel.egi.cesga.es
REG_HOST=lcgic01.gridpp.rl.ac.uk
SITE_BDII_HOST=sitebdii.egi.cesga.es
YAIM_LOGGING_LEVEL=DEBUG
#####################################
# CREAM CE node cluster mode #
#####################################
## Be sure that CREAMCE_CLUSTER_MODE is set to no (or not set at all).
## CREAMCE_CLUSTER_MODE="no"
#####################################
# CREAM CE Variables #
#####################################
CEMON_HOST=cream.egi.cesga.es
CREAM_DB_USER=""
CREAM_DB_PASSWORD=""
MYSQL_PASSWORD=""
BLPARSER_HOST=cream.egi.cesga.es
##The CREAM sandbox directory name (default /var/cream_sandbox) can be changed using the yaim variable CREAM_SANDBOX_DIR".
CREAM_SANDBOX_DIR="/home_grid/var/cream_sandbox"
CREAM_SANDBOX_PATH="/home_grid/var/cream_sandbox"
########################################
# SGE configuration variables #
########################################
BATCH_SERVER="svgd.cesga.es"
## Jobmanager specific settings
BLPARSER_WITH_UPDATER_NOTIFIER="true"
JOB_MANAGER="sge"
CE_BATCH_SYS="sge"
SGE_ROOT="/usr/local/sge/pro"
SGE_CELL="default"
## GE MASTER PORTS
SGE_QMASTER=6448
SGE_EXECD=6449
SGE_SPOOL_METH="classic"
SGE_CLUSTER_NAME="svgd"
SGE_SHARED_INSTALL="yes"
BATCH_LOG_DIR="/usr/local/sge/pro/default/common"
BATCH_BIN_DIR="/usr/local/sge/pro/bin/lx26-amd64"
BATCH_VERSION="sge-6.2u5"
################################
# APEL configuration variables #
################################
APEL_MYSQL_HOST=apel.egi.cesga.es
APEL_DB_PASSWORD=""
################################
# MPI configuration variables #
################################
MPI_MPICH2_ENABLE="yes"
MPI_MPICH2_VERSION="1.3.2p1"
# If you do NOT provide a shared home, set $MPI_SHARED_HOME to "no" (default).
MPI_SHARED_HOME="yes"
# These variables tell YAIM where to find additional configuration files.
WN_LIST=/opt/glite/yaim/etc/wn-list.conf
USERS_CONF=/opt/glite/yaim/etc/users.conf
GROUPS_CONF=/opt/glite/yaim/etc/groups.conf
FUNCTIONS_DIR=/opt/glite/yaim/functions
# Set this to your prefered and firewall allowed port range
GLOBUS_TCP_PORT_RANGE="20000,25000"
# Site-wide settings
SITE_EMAIL=grid-admin@cesga.es
#New yaim 3.0.1
SITE_SUPPORT_EMAIL=$SITE_EMAIL
SITE_NAME=CESGA
SITE_LOC="Santiago de Compostela, Spain"
SITE_LAT=42.875558 #42.8757 # -90 to 90 degrees
SITE_LONG=-8.553147 #-8.5536 # -180 to 180 degrees
SITE_WEB="http://www.egee.cesga.es"
# Architecture and enviroment specific settings
CE_CPU_MODEL=Opteron
CE_CPU_VENDOR=amd
CE_CPU_SPEED=2200
CE_OS="ScientificSL" # Forma correcta
CE_OS_RELEASE=6.0
CE_OS_VERSION="Carbon"
#New variable
CE_PHYSCPU=30
CE_LOGCPU=720
CE_OS_ARCH=x86_64
CE_CAPABILITY="CPUScalingReferenceSI00=2395"
CE_OTHERDESCR="Cores=24,Benchmark=9.58-HEP-SPEC06"
SE_MOUNT_INFO_LIST="none"
CE_SI00=2395
CE_MINPHYSMEM=1524
CE_MINVIRTMEM=512
CE_SMPSIZE=2
CE_SF00=1714
CE_OUTBOUNDIP=TRUE
CE_INBOUNDIP=FALSE
CE_RUNTIMEENV="
LCG-2
LCG-2_1_0
LCG-2_1_1
LCG-2_2_0
LCG-2_3_0
LCG-2_3_1
LCG-2_4_0
LCG-2_5_0
LCG-2_6_0
LCG-2_7_0
GLITE-3_0_0
GLITE-3_0_2
GLITE-3_1_0
R-GMA
"
################################
# SE/DPM-MySQL configuration variables #
################################
# The name of the DPM head node
DPM_HOST=se2.egi.cesga.es
# The DPM pool name
#DPMPOOL=the_dpm_pool_name
DPMPOOL=egee-pool
# The filesystems/partitions parts of the pool
#DPM_FILESYSTEMS="$DPM_HOST:/path1 my-dpm-poolnode.$MY_DOMAIN:/path2"
DPM_FILESYSTEMS="$DPM_HOST:/storage $DPM_HOST:/mnt/xen01"
# The database user
#DPM_DB_USER=the-dpm-db-user
DPM_DB_USER=dpm_user
# The database user password
#DPM_DB_PASSWORD=the-dpm-db-user-pwd
DPM_DB_PASSWORD=
# The DPM database host
#DPM_DB_HOST=$DPM_HOST
DPM_DB_HOST=$DPM_HOST
# Specifies the default amount of space reserved for a file
DPMFSIZE=200M
DPM_INFO_USER=dpm_info
DPM_INFO_PASS=
# This largely replaces CE_CLOSE_SE but it is a list of hostnames
SE_LIST="$DPM_HOST"
SE_ARCH="multidisk" # "disk, tape, multidisk, other"
## The following lines should not be neccesary, I just put them, but I think they are not used -- esfreire 16022012
#
# BDII/GIP specific settings
BDII_HTTP_URL="http://grid-deployment.web.cern.ch/grid-deployment/gis/lcg2-bdii/dteam/lcg2-all-sites.conf"
# Set this to use FCR
BDII_FCR="http://goc.grid-support.ac.uk/gridsite/bdii/BDII/www/bdii-update.ldif"
# Ex.: BDII_REGIONS="CE SE RB PX VOBOX"
# List of nodes included into CESGA SITE:
BDII_REGIONS="CE2 SE MON LFC WMS PX NAGIOS VOMS BDII TOPBDII" # list of the services provided by the site
BDII_CE2_URL="ldap://$CE_HOST:2170/mds-vo-name=resource,o=grid"
BDII_SE_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid"
BDII_RB_URL="ldap://$RB_HOST:2135/mds-vo-name=local,o=grid"
BDII_MON_URL="ldap://$MON_HOST:2170/mds-vo-name=resource,o=grid"
BDII_PX_URL="ldap://$PX_HOST:2170/mds-vo-name=resource,o=grid"
BDII_LFC_URL="ldap://lfc.egi.cesga.es:2170/mds-vo-name=resource,o=grid"
BDII_WMS_URL="ldap://$WMS_HOST:2170/mds-vo-name=resource,o=grid"
BDII_FTS_URL="ldap://$FTS_HOST:2170/mds-vo-name=resource,o=grid"
BDII_NAGIOS_URL="ldap://rnagios.ibergrid.cesga.es:2170/mds-vo-name=resource,o=grid"
BDII_VOMS_URL="ldap://voms.egi.cesga.es:2170/mds-vo-name=resource,o=grid"
BDII_BDII_URL="ldap://sitebdii.egi.cesga.es:2170/mds-vo-name=resource,o=grid"
BDII_TOPBDII_URL="ldap://topbdii.egi.cesga.es:2170/mds-vo-name=resource,o=grid"
# VOS="atlas alice lhcb cms dteam biomed"
# Space separated list of supported VOs by your site
VOS="ops cesga dteam biomed compchem fusion imath.cesga.es ops.vo.ibergrid.eu iber.vo.ibergrid.eu tut.vo.ibergrid.eu eng.vo.ibergrid.eu ict.vo.ibergrid.eu life.vo.ibergrid.eu hpc.vo.ibergrid.eu earth.vo.ibergrid.
eu bing.vo.ibergrid.eu pfound.vo.ibergrid.eu phys.vo.ibergrid.eu social.vo.ibergrid.eu chem.vo.ibergrid.eu auger mpi"
QUEUES="GRIDEGI_ops GRIDEGI_large GRIDEGI_small GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_small GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_la
rge GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large GRIDEGI_large"
VO_SW_DIR="/opt/exp_soft2"
#New in Yaim 3.0.1
GRIDEGI_OPS_GROUP_ENABLE="ops /VO=ops/GROUP=/ops/ROLE=lcgadmin"
GRIDEGI_SMALL_GROUP_ENABLE="dteam /VO=dteam/GROUP=/dteam/ROLE=lcgadmin ops.vo.ibergrid.eu /VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=VO-Admin /VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=Produ
ction"
GRIDEGI_LARGE_GROUP_ENABLE="cesga /VO=cesga/GROUP=/cesga/ROLE=VO-Admin biomed /VO=biomed/GROUP=/biomed/ROLE=lcgadmin compchem /VO=compchem/GROUP=/compchem/ROLE=lcgadmin fusion /VO=fusion/GROUP=/fusion/ROLE=VO-ad
min imath.cesga.es /VO=imath.cesga.es/GROUP=/imath.cesga.es/ROLE=VO-Admin iber.vo.ibergrid.eu /VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/ROLE=VO-Admin /VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/RO
LE=Production tut.vo.ibergrid.eu /VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=VO-Admin /VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=Production eng.vo.ibergrid.eu /VO=eng.vo.ibergrid.eu/GROUP=/en
g.vo.ibergrid.eu/ROLE=VO-Admin /VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu/ROLE=Production ict.vo.ibergrid.eu /VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu/ROLE=VO-Admin /VO=ict.vo.ibergrid.eu/GROUP=/ict.
vo.ibergrid.eu/ROLE=Production life.vo.ibergrid.eu /VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=VO-Admin /VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=Production hpc.vo.ibergrid.eu /VO=hpc.vo
.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=VO-Admin /VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=Production earth.vo.ibergrid.eu /VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=VO-Admin /VO=ea
rth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=Production bing.vo.ibergrid.eu /VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=VO-Admin /VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=Producti
on pfound.vo.ibergrid.eu /VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=VO-Admin /VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=Production phys.vo.ibergrid.eu /VO=phys.vo.ibergrid.eu/GRO
UP=/phys.vo.ibergrid.eu/ROLE=VO-Admin /VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu/ROLE=Production social.vo.ibergrid.eu /VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=VO-Admin /VO=social.vo.ib
ergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=Production chem.vo.ibergrid.eu /VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=VO-Admin /VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=Production auger
/VO=auger/GROUP=/auger/ROLE=SoftwareManager /VO=auger/GROUP=/auger/ROLE=Production /VO=auger/GROUP=/auger/FNAL mpi /VO=mpi/GROUP=/mpi/ROLE=SoftwareManager /VO=mpi/GROUP=/mpi/ROLE=Production"
# VO specific settings. For help see: https://lcg-sft.cern.ch/yaimtool/yaimtool.py
#::::::::::::::
#auger
#::::::::::::::
VO_AUGER_SW_DIR=$VO_SW_DIR/augersgm
VO_AUGER_DEFAULT_SE=$DPM_HOST
VO_AUGER_STORAGE_DIR=$CLASSIC_STORAGE_DIR/auger
VO_AUGER_QUEUES="GRIDEGI_large"
VO_AUGER_VOMS_SERVERS='vomss://voms1.egee.cesnet.cz:8443/voms/auger'
VO_AUGER_VOMSES="'auger voms1.egee.cesnet.cz 15004 /DC=org/DC=terena/DC=tcs/C=CZ/O=CESNET/CN=voms1.egee.cesnet.cz auger 24'"
VO_AUGER_VOMS_CA_DN="'/C=NL/O=TERENA/CN=TERENA eScience SSL CA'"
#::::::::::::::
#mpi
#::::::::::::::
VO_MPI_SW_DIR=$VO_SW_DIR/mpisgm
VO_MPI_DEFAULT_SE=$DPM_HOST
VO_MPI_STORAGE_DIR=$CLASSIC_STORAGE_DIR/mpi
VO_MPI_QUEUES="GRIDEGI_large"
VO_MPI_VOMS_SERVERS='vomss://voms1.egee.cesnet.cz:8443/voms/mpi?/mpi'
VO_MPI_VOMSES="'mpi voms1.egee.cesnet.cz 15030 /DC=cz/DC=cesnet-ca/O=CESNET/CN=voms1.egee.cesnet.cz mpi 24'"
VO_MPI_VOMS_CA_DN="'/DC=cz/DC=cesnet-ca/O=CESNET CA/CN=CESNET CA 3'"
VO_MPI_WMS_HOSTS="wms1.egee.cesnet.cz wms2.egee.cesnet.cz"
#::::::::::::::
#biomed
#::::::::::::::
VO_BIOMED_SW_DIR=$VO_SW_DIR/biomed
VO_BIOMED_DEFAULT_SE=$DPM_HOST
VO_BIOMED_STORAGE_DIR=$CLASSIC_STORAGE_DIR/biomed
VO_BIOMED_QUEUES="GRIDEGI_large"
VO_BIOMED_VOMS_SERVERS="'vomss://cclcgvomsli01.in2p3.fr:8443/voms/biomed'"
VO_BIOMED_VOMSES="'biomed cclcgvomsli01.in2p3.fr 15000 /O=GRID-FR/C=FR/O=CNRS/OU=CC-IN2P3/CN=cclcgvomsli01.in2p3.fr biomed'"
VO_BIOMED_VOMS_CA_DN="'/C=FR/O=CNRS/CN=GRID2-FR'"
#::::::::::::::
#cesga
#::::::::::::::
VO_CESGA_SW_DIR=$VO_SW_DIR/cesga
VO_CESGA_DEFAULT_SE=$DPM_HOST
VO_CESGA_STORAGE_DIR=$CLASSIC_STORAGE_DIR/cesga
VO_CESGA_QUEUES="GRIDEGI_large"
VO_CESGA_USERS=vomss://voms.egi.cesga.es:8443/voms/cesga?/cesga
VO_CESGA_VOMS_SERVERS="'vomss://voms.egi.cesga.es:8443/voms/cesga?/cesga'"
VO_CESGA_VOMSES="'cesga voms.egi.cesga.es 15110 /DC=es/DC=irisgrid/O=cesga/CN=host/voms.egi.cesga.es cesga'"
VO_CESGA_VOMS_CA_DN="'/DC=es/DC=irisgrid/CN=IRISGridCA'"
#::::::::::::::
#compchem
#::::::::::::::
VO_COMPCHEM_SW_DIR=$VO_SW_DIR/compchem
VO_COMPCHEM_DEFAULT_SE=$DPM_HOST
VO_COMPCHEM_STORAGE_DIR=$CLASSIC_STORAGE_DIR/compchem
VO_COMPCHEM_QUEUES="GRIDEGI_large"
VO_COMPCHEM_USERS=vomss://voms.cnaf.infn.it:8443/voms/compchem?/compchem
VO_COMPCHEM_VOMS_SERVERS="'vomss://voms.cnaf.infn.it:8443/voms/compchem?/compchem' 'vomss://voms-01.pd.infn.it:8443/voms/compchem?/compchem'"
VO_COMPCHEM_VOMSES="'compchem voms.cnaf.infn.it 15003 /C=IT/O=INFN/OU=Host/L=CNAF/CN=voms.cnaf.infn.it compchem' 'compchem voms-01.pd.infn.it 15003 /C=IT/O=INFN/OU=Host/L=Padova/CN=voms-01.pd.infn.it compchem'"
VO_COMPCHEM_VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN CA' '/C=IT/O=INFN/CN=INFN CA'"
#::::::::::::::
#dteam
#::::::::::::::
VO_DTEAM_SW_DIR=$VO_SW_DIR/dteam
VO_DTEAM_DEFAULT_SE=$DPM_HOST
VO_DTEAM_STORAGE_DIR=$CLASSIC_STORAGE_DIR/dteam
VO_DTEAM_QUEUES="GRIDEGI_small"
VO_DTEAM_VOMS_SERVERS='vomss://voms.hellasgrid.gr:8443/voms/dteam?/dteam/'
VO_DTEAM_VOMSES="'dteam lcg-voms.cern.ch 15004 /DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch dteam 24' 'dteam voms.cern.ch 15004 /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch dteam 24' 'dteam voms.hellasgrid.gr 1
5004 /C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr dteam 24' 'dteam voms2.hellasgrid.gr 15004 /C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr dteam 24'"
VO_DTEAM_VOMS_CA_DN="'/DC=ch/DC=cern/CN=CERN Trusted Certification Authority' '/DC=ch/DC=cern/CN=CERN Trusted Certification Authority' '/C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2006' '/C=G
R/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2006'"
#::::::::::::::
#fusion
#::::::::::::::
VO_FUSION_SW_DIR=$VO_SW_DIR/fusion
VO_FUSION_DEFAULT_SE=$DPM_HOST
VO_FUSION_STORAGE_DIR=$CLASSIC_STORAGE_DIR/fusion
VO_FUSION_QUEUES="GRIDEGI_large"
VO_FUSION_VOMS_SERVERS="vomss://voms-prg.bifi.unizar.es:8443/voms/fusion?/fusion/"
VO_FUSION_VOMSES="'fusion voms-prg.bifi.unizar.es 15001 /DC=es/DC=irisgrid/O=bifi-unizar/CN=voms-prg.bifi.unizar.es fusion'"
VO_FUSION_VOMS_CA_DN="'/DC=es/DC=irisgrid/CN=IRISGridCA'"
#::::::::::::::
#imath.cesga.es
#::::::::::::::
VO_IMATH_CESGA_ES_SW_DIR=$VO_SW_DIR/imath
VO_IMATH_CESGA_ES_DEFAULT_SE=$DPM_HOST
VO_IMATH_CESGA_ES_STORAGE_DIR=$CLASSIC_STORAGE_DIR/imath
VO_IMATH_CESGA_ES_QUEUES="GRIDEGI_large"
VO_IMATH_CESGA_ES_USERS=vomss://voms.egi.cesga.es:8443/voms/imath.cesga.es?/imath.cesga.es
VO_IMATH_CESGA_ES_VOMS_SERVERS="'vomss://voms.egi.cesga.es:8443/voms/imath.cesga.es?/imath.cesga.es'"
VO_IMATH_CESGA_ES_VOMSES="'imath.cesga.es voms.egi.cesga.es 15100 /DC=es/DC=irisgrid/O=cesga/CN=host/voms.egi.cesga.es imath.cesga.es'"
VO_IMATH_CESGA_ES_VOMS_CA_DN="'/DC=es/DC=irisgrid/CN=IRISGridCA'"
#::::::::::::::
#ops
#::::::::::::::
VO_OPS_SW_DIR=$VO_SW_DIR/ops
VO_OPS_DEFAULT_SE=$DPM_HOST
VO_OPS_STORAGE_DIR=$CLASSIC_STORAGE_DIR/ops
VO_OPS_QUEUES="GRIDEGI_ops"
VO_OPS_VOMS_SERVERS="'vomss://voms.cern.ch:8443/voms/ops?/ops/'"
VO_OPS_VOMSES="'ops voms.cern.ch 15009 /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch ops' 'ops lcg-voms.cern.ch 15009 /DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch ops'"
VO_OPS_VOMS_CA_DN="'/DC=ch/DC=cern/CN=CERN Trusted Certification Authority' '/DC=ch/DC=cern/CN=CERN Trusted Certification Authority'"
## IBERGRID VOS ##
VO_LIFE_VO_IBERGRID_EU_VOMSES="'life.vo.ibergrid.eu voms01.ncg.ingrid.pt 40010 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt life.vo.ibergrid.eu' 'life.vo.ibergrid.eu ibergrid-voms.ifca.es 40010 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es life.vo.ibergrid.eu'"
VO_LIFE_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# hpc.vo.ibergrid.eu
VO_HPC_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/hpcibeusgm
VO_HPC_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_HPC_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/hpcibeu
VO_HPC_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_HPC_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/hpc.vo.ibergrid.eu?/hpc.vo.ibergrid.eu"
VO_HPC_VO_IBERGRID_EU_VOMSES="'hpc.vo.ibergrid.eu voms01.ncg.ingrid.pt 40005 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt hpc.vo.ibergrid.eu' 'hpc.vo.ibergrid.eu ibergrid-voms.ifca.es 40005 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es hpc.vo.ibergrid.eu'"
VO_HPC_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# earth.vo.ibergrid.eu
VO_EARTH_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/earibeusgm
VO_EARTH_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_EARTH_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/earibeu
VO_EARTH_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_EARTH_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/earth.vo.ibergrid.eu?/earth.vo.ibergrid.eu"
VO_EARTH_VO_IBERGRID_EU_VOMSES="'earth.vo.ibergrid.eu voms01.ncg.ingrid.pt 40011 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt earth.vo.ibergrid.eu' 'earth.vo.ibergrid.eu ibergrid-voms.ifca.es 40011 /DC=e
s/DC=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es earth.vo.ibergrid.eu'"
VO_EARTH_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# bing.vo.ibergrid.eu
VO_BING_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/binibeusgm
VO_BING_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_BING_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/binibeu
VO_BING_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_BING_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/bing.vo.ibergrid.eu?/bing.vo.ibergrid.eu"
VO_BING_VO_IBERGRID_EU_VOMSES="'bing.vo.ibergrid.eu voms01.ncg.ingrid.pt 40004 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt bing.vo.ibergrid.eu' 'bing.vo.ibergrid.eu ibergrid-voms.ifca.es 40004 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es bing.vo.ibergrid.eu'"
VO_BING_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# pfound.vo.ibergrid.eu
VO_PFOUND_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/pfoibeusgm
VO_PFOUND_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_PFOUND_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/pfoibeu
VO_PFOUND_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_PFOUND_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/pfound.vo.ibergrid.eu?/pfound.vo.ibergrid.eu"
VO_PFOUND_VO_IBERGRID_EU_VOMSES="'pfound.vo.ibergrid.eu voms01.ncg.ingrid.pt 40006 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt pfound.vo.ibergrid.eu' 'pfound.vo.ibergrid.eu ibergrid-voms.ifca.es 40006 /
DC=es/DC=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es pfound.vo.ibergrid.eu'"
VO_PFOUND_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# phys.vo.ibergrid.eu
VO_PHYS_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/phyibeusgm
VO_PHYS_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_PHYS_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/phyibeu
VO_PHYS_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_PHYS_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/phys.vo.ibergrid.eu?/phys.vo.ibergrid.eu"
VO_PHYS_VO_IBERGRID_EU_VOMSES="'phys.vo.ibergrid.eu voms01.ncg.ingrid.pt 40007 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt phys.vo.ibergrid.eu' 'phys.vo.ibergrid.eu ibergrid-voms.ifca.es 40007 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es phys.vo.ibergrid.eu'"
VO_PHYS_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# social.vo.ibergrid.eu
VO_SOCIAL_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/socibeusgm
VO_SOCIAL_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
# ops.vo.ibergrid.eu
VO_OPS_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/opsibeusgm
VO_OPS_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_OPS_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/opsibeu
VO_OPS_VO_IBERGRID_EU_QUEUES="GRIDEGI_small"
VO_OPS_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/ops.vo.ibergrid.eu?/ops.vo.ibergrid.eu'"
VO_OPS_VO_IBERGRID_EU_VOMSES="'ops.vo.ibergrid.eu voms01.ncg.ingrid.pt 40001 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt ops.vo.ibergrid.eu' 'ops.vo.ibergrid.eu ibergrid-voms.ifca.es 40001 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es ops.vo.ibergrid.eu'"
VO_OPS_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# tut.vo.ibergrid.eu
VO_TUT_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/tutibeusgm
VO_TUT_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_TUT_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/tutibeu
VO_TUT_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_TUT_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/tut.vo.ibergrid.eu?/tut.vo.ibergrid.eu'"
VO_TUT_VO_IBERGRID_EU_VOMSES="'tut.vo.ibergrid.eu voms01.ncg.ingrid.pt 40002 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt tut.vo.ibergrid.eu' 'tut.vo.ibergrid.eu ibergrid-voms.ifca.es 40002 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es tut.vo.ibergrid.eu'"
VO_TUT_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# iber.vo.ibergrid.eu
VO_IBER_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/iberibeusgm
VO_IBER_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_IBER_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/iberibeu
VO_IBER_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_IBER_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/iber.vo.ibergrid.eu?/iber.vo.ibergrid.eu'"
VO_IBER_VO_IBERGRID_EU_VOMSES="'iber.vo.ibergrid.eu voms01.ncg.ingrid.pt 40003 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt iber.vo.ibergrid.eu' 'iber.vo.ibergrid.eu ibergrid-voms.ifca.es 40003 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es iber.vo.ibergrid.eu'"
VO_IBER_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# eng.vo.ibergrid.eu
VO_ENG_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/engibeusgm
VO_ENG_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_ENG_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/engibeu
VO_ENG_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_ENG_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/eng.vo.ibergrid.eu?/eng.vo.ibergrid.eu"
VO_ENG_VO_IBERGRID_EU_VOMSES="'eng.vo.ibergrid.eu voms01.ncg.ingrid.pt 40013 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt eng.vo.ibergrid.eu' 'eng.vo.ibergrid.eu ibergrid-voms.ifca.es 40013 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es eng.vo.ibergrid.eu'"
VO_ENG_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# ict.vo.ibergrid.eu
VO_ICT_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/ictibeusgm
VO_ICT_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_ICT_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/ictibeu
VO_ICT_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_ICT_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/ict.vo.ibergrid.eu?/ict.vo.ibergrid.eu"
VO_ICT_VO_IBERGRID_EU_VOMSES="'ict.vo.ibergrid.eu voms01.ncg.ingrid.pt 40008 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt ict.vo.ibergrid.eu' 'ict.vo.ibergrid.eu ibergrid-voms.ifca.es 40008 /DC=es/DC=iri
sgrid/O=ifca/CN=host/ibergrid-voms.ifca.es ict.vo.ibergrid.eu'"
VO_ICT_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# life.vo.ibergrid.eu
VO_LIFE_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/lifibeusgm
VO_LIFE_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_LIFE_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/lifibeu
VO_LIFE_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_LIFE_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/life.vo.ibergrid.eu?/life.vo.ibergrid.eu"
VO_SOCIAL_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/socibeu
VO_SOCIAL_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_SOCIAL_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/social.vo.ibergrid.eu"?/social.vo.ibergrid.eu""
VO_SOCIAL_VO_IBERGRID_EU_VOMSES="'social.vo.ibergrid.eu voms01.ncg.ingrid.pt 40012 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt social.vo.ibergrid.eu' 'social.vo.ibergrid.eu ibergrid-voms.ifca.es 40012 /
DC=es/DC=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es social.vo.ibergrid.eu'"
VO_SOCIAL_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"
# chem.vo.ibergrid.eu
VO_CHEM_VO_IBERGRID_EU_SW_DIR=$VO_SW_DIR/cheibeusgm
VO_CHEM_VO_IBERGRID_EU_DEFAULT_SE=$DPM_HOST
VO_CHEM_VO_IBERGRID_EU_STORAGE_DIR=$CLASSIC_STORAGE_DIR/cheibeu
VO_CHEM_VO_IBERGRID_EU_QUEUES="GRIDEGI_large"
VO_CHEM_VO_IBERGRID_EU_VOMS_SERVERS="'vomss://voms01.ncg.ingrid.pt:8443/voms/chem.vo.ibergrid.eu"?/chem.vo.ibergrid.eu""
VO_CHEM_VO_IBERGRID_EU_VOMSES="'chem.vo.ibergrid.eu voms01.ncg.ingrid.pt 40009 /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt chem.vo.ibergrid.eu' 'chem.vo.ibergrid.eu ibergrid-voms.ifca.es 40009 /DC=es/DC
=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es chem.vo.ibergrid.eu'"
VO_CHEM_VO_IBERGRID_EU_VOMS_CA_DN="'/C=PT/O=LIPCA/CN=LIP Certification Authority' '/DC=es/DC=irisgrid/CN=IRISGridCA'"- Groups.conf file used
"/VO=cesga/GROUP=/cesga/ROLE=VO-Admin":::sgm: "/VO=cesga/GROUP=/cesga":::: "/VO=compchem/GROUP=/compchem/ROLE=SoftwareManager":::sgm: "/VO=compchem/GROUP=/compchem":::: "/VO=fusion/GROUP=/fusion/ROLE=VO-admin":::sgm: "/VO=fusion/GROUP=/fusion/ROLE=swadmin":::sgm: "/VO=fusion/GROUP=/fusion/ROLE=production":::prd: "/VO=fusion/GROUP=/fusion":::: "/VO=imath.cesga.es/GROUP=/imath.cesga.es/ROLE=VO-Admin":::sgm: "/VO=imath.cesga.es/GROUP=/imath.cesga.es":::: "/dteam/ROLE=lcgadmin":::sgm: "/dteam/ROLE=production":::prd: "/dteam":::: "/auger/ROLE=SoftwareManager":::sgm: "/auger/ROLE=Production":::prd: "/auger":::: "/auger/FNAL":::: "/mpi/ROLE=SoftwareManager":::sgm: "/mpi/ROLE=Production":::prd: "/mpi":::: "/biomed/ROLE=lcgadmin":::sgm: "/biomed/ROLE=production":::prd: "/biomed":::: "/ops/ROLE=lcgadmin":::sgm: "/ops":::: "/VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=ops.vo.ibergrid.eu/GROUP=/ops.vo.ibergrid.eu":::: "/VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=iber.vo.ibergrid.eu/GROUP=/iber.vo.ibergrid.eu":::: "/VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=tut.vo.ibergrid.eu/GROUP=/tut.vo.ibergrid.eu":::: "/VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=eng.vo.ibergrid.eu/GROUP=/eng.vo.ibergrid.eu":::: "/VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=ict.vo.ibergrid.eu/GROUP=/ict.vo.ibergrid.eu":::: "/VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=life.vo.ibergrid.eu/GROUP=/life.vo.ibergrid.eu"::: "/VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=hpc.vo.ibergrid.eu/GROUP=/hpc.vo.ibergrid.eu":::: "/VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=earth.vo.ibergrid.eu/GROUP=/earth.vo.ibergrid.eu":::: "/VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=bing.vo.ibergrid.eu/GROUP=/bing.vo.ibergrid.eu":::: "/VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=pfound.vo.ibergrid.eu/GROUP=/pfound.vo.ibergrid.eu":::: "/VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=phys.vo.ibergrid.eu/GROUP=/phys.vo.ibergrid.eu":::: "/VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=social.vo.ibergrid.eu/GROUP=/social.vo.ibergrid.eu":::: "/VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=VO-Admin":::sgm: "/VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=SW-Admin":::sgm: "/VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu/ROLE=Production":::prd: "/VO=chem.vo.ibergrid.eu/GROUP=/chem.vo.ibergrid.eu"::::
- Before configuring with YAIM, we decide to share the Users HOMES between the CreamCE and the WNS
Sharing the GRID HOMES mounted front the EMC storage server with the WNs
[root@cream ~]# cat /etc/fstab # # /etc/fstab # Created by anaconda on Mon Mar 14 16:30:46 2011 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # UUID=afb1e256-6b16-493b-9347-48737b03a990 / ext3 defaults 1 1 /dev/sda swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 # GRID HOMES (Shared between the WNs and the new CreamCE(EGI)) 193.144.35.13:/Grid_se/home_grid /mnt/EMC/Grid_se-home_grid nfs rw,rsize=8192,wsize=8192,intr,noatime,vers=3,tcp 0 0 # OPT_CESGA_SVG (in order to read the accounting file from SVG in real time) 193.144.35.13:/Optcesga_SVG /mnt/EMC/Optcesga_SVG nfs rw,rsize=8192,wsize=8192,intr,noatime,vers=3,tcp 0 0
We had to set up NFS version 3, because we got problems with NFS version 4, if we tried to execute a chown of a file we got "invalid argument"
- Creating the necessary files before mounting the directories
mkdir /mnt/EMC/ mkdir /mnt/EMC/Grid_se-home_grid ln -sf /mnt/EMC/Grid_se-home_grid /home_grid
- Install necessary packages to mount NFS
yum install nfs-utils.x86_64 yum install rpc2
- Changing where useradd will create the home for a User
[root@cream home_grid]# cp -aprx /etc/default/useradd /etc/default/useradd.orig [root@cream home_grid]# cat /etc/default/useradd # useradd defaults file GROUP=100 HOME=/home_grid INACTIVE=-1 EXPIRE= SHELL=/bin/bash SKEL=/etc/skel CREATE_MAIL_SPOOL=yes
Configuring with YAIM (OK)
/opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n MPI_CE -n creamCE -n SGE_utils [ .... ]
Configuring IPTABLES on CreamCE
Open the following ports in the CreamCe Iptables according to the following link, https://wiki.italiangrid.it/twiki/bin/view/CREAM/ServiceReferenceCard#Open_ports
#CREAM -A RH-Firewall-1-INPUT -p tcp --dport 8443 -j ACCEPT #GSIFTP -A RH-Firewall-1-INPUT -p tcp --dport 2811 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp --dport 9002 -j ACCEPT #BDII (EL SITE BDII ESTA EN EL MON) -A RH-Firewall-1-INPUT -p tcp --dport 2170 -j ACCEPT
Creating the SGE enviornment on the CreamCE
- Creating a file by hand in /etc/profile.d to load SGE environment on the CreamCE
[root@cream profile.d]# ll | grep -i sge -rw-r--r-- 1 root root 152 Feb 17 09:56 sge.sh [root@cream profile.d]# cat /etc/profile.d/sge.sh # Define SGE_ROOT directory and SGE commands export SGE_ROOT=/usr/local/sge/pro export SGE_CELL=default . /usr/local/sge/pro/default/common/settings.sh
Mounting the accounting file from SVG
In order to see the accounting file in real time, we have mounted the /opt/cesga from SVG as Read-only file system, and we have created a symbolic link in the local SGE installation in the CreamCE to the accounting file from SVG. This is necessary on the CreamCE service to have information about job status in real time (executed, cancelled, suspended, finished, etc.)
[root@cream bakcup_cream]# cat /etc/fstab [ ...... ] # OPT_CESGA_SVG (in order to read the accounting file from SVG in real time) 193.144.35.13:/Optcesga_SVG /mnt/EMC/Optcesga_SVG nfs rw,rsize=8192,wsize=8192,intr,noatime,nfsvers=3,tcp 0 0 [root@cream home_grid]# mkdir /mnt/EMC/Optcesga_SVG [root@cream common]# ln -s /mnt/EMC/Optcesga_SVG/opt/cesga/sge/default/common/accounting /usr/local/sge/pro/default/common/accounting
Configuring the queues and the File transfers
SEE the following link, https://wiki.italiangrid.it/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#1_5_3_5_1_File_transfers
- We have created the following directory, /home_grid/var/cream_sandbox, which is mounted/shared in all the GRID_WNs, besides, we have configured the following variable in the site-info.def
##The CREAM sandbox directory name (default /var/cream_sandbox) can be changed using the yaim variable CREAM_SANDBOX_DIR". CREAM_SANDBOX_DIR="/home_grid/var/cream_sandbox"
- After the YAIM configuration the following file is changed:
[root@cream ~]# grep -i sandbox_dir /etc/glite-ce-cream/cream-config.xml
<parameter name="CREAM_SANDBOX_DIR" value="/home_grid/var/cream_sandbox" />
<parameter name="CREAM_SANDBOX_DIR" value="/home_grid/var/cream_sandbox" />- We needed to create three new queues:
GRIDEGI_large GRIDEGI_ops GRIDEGI_small
- And in order to configure the file transfers, we needed to configure a prolog/epilog on all these queues:
prolog root@/opt/exp_soft2/SGE_prolog/prolog.sh epilog /opt/exp_soft2/SGE_prolog/sge_filestaging --stageout
- Where /opt/exp_soft2/SGE_prolog/prolog.sh contents:
[root@compute-5-1 exp_soft2]# cat /opt/exp_soft2/SGE_prolog/prolog.sh #!/bin/bash chown $USER $JOB_SCRIPT chgrp `id $USER -gn` $JOB_SCRIPT chmod 700 $JOB_SCRIPT su -m -c "/opt/exp_soft2/SGE_prolog/sge_filestaging --stagein" $USER exit 0
- And /opt/exp_soft2/SGE_prolog/sge_filestaging contents:
[root@compute-5-1 exp_soft2]# cat /opt/exp_soft2/SGE_prolog/sge_filestaging
#! /usr/bin/perl
use Getopt::Long;
my ( $STAGEIN, $FILELIST );
GetOptions(
'stagein' => sub { $STAGEIN = 1; $FILELIST = 'SGE_stagein'; },
'stageout' => sub { $STAGEIN = 0; $FILELIST = 'SGE_stageout'; },
);
exit 0 unless $FILELIST;
my ( @FILES ) = split( /\@\@\@/, $ENV{$FILELIST} );
foreach ( @FILES ) {
next unless /^([^\@]+)\@([^\:]+)\:(.+)$/;
my $localfile = $1;
my $remotemachine = $2;
my $remotefile = $3;
if ( $STAGEIN ) {
system( 'cp', $remotefile, $localfile );
} else {
system( 'cp', $localfile, $remotefile );
}
}
exit( 0 ); The directory /opt/exp_soft2/SGE_prolog was created by hand in order to this directory is being seen in all the WNs
Assigning complex to the jobs and modifying the sge_submit.sh script for it
- We added the following lines to the /usr/libexec/sge_submit.sh script:
. /etc/lrms/ge_globals.conf #(added on line 37)
local_submit_attributes_file=${GLITE_LOCATION:-/opt/glite}/libexec/sge_local_submit_attributes.sh #(added on line 74)
# Write GE directives according to /etc/lrms/ge_local_submit_wrapper.conf #(added on line 102)
ge_retcode=`$GE_REQS_SCRIPT --user-attrs=${bls_opt_req_file} $bls_opt_queue $bls_tmp_file 2>&1` #(added on line 103)
[ '$ge_retcode' != '0' ] && echo "$ge_retcode" 1>&2 #(added on line 104)- Where:
- /etc/lrms/ge_globals.conf file is used to load the /usr/libexec/ge_local_submit_wrapper script to parser the complex from the /etc/lrms/ge_local_submit_wrapper.conf file (in which, we configure the complex that we want to assign to the jobs). Its content is:
[root@cream ~]# cat /etc/lrms/ge_globals.conf GE_SHARED_SANDBOX=yes GE_REQS_SCRIPT=/usr/libexec/ge_local_submit_wrapper
- It is necessary modify the /usr/libexec/ge_local_submit_wrapper script to configure /etc/lrms/ge_local_submit_wrapper.conf file. Modify the line 10 to add the following content to the variable filename:
filename = '/etc/lrms/ge_local_submit_wrapper.conf'
- Configuring the respective complex for the submitted jobs, these complex are added to the job when it is submitted:
[root@cream ~]# cat /etc/lrms/ge_local_submit_wrapper.conf *|complex_values|num_proc=1,h_stack=32M,arch=x86_64,processor=opteron_6174,h_fsize=10G *|s_rt|150:00:00 *|s_vmem|1G GRIDEGI_ops|s_vmem|4G
The '*' means that the complex are applied to all the jobs, but it is possible to substitute the '*' for the name of the queue to which we want to apply the complex. WARNING: The "-cwd" option must not be NEVER used, because otherwise it takes the /var/tmp directory as HOME because the qsub is executed from the /var/tmp directory, and "-cwd" option what it does is to use current working directory.
- This is the content for /usr/libexec/sge_submit.sh script once modified:
[root@cream ~]# cat /usr/libexec/sge_submit.sh
#!/bin/bash
#
# File: sge_submit.sh
# Author: Keith Sephton (kms@doc.ic.ac.uk)
#
# Based on pbs_submit.sh
# Author: David Rebatto (david.rebatto@mi.infn.it)
#
# Revision history:
# xx-Apr-2008: Original release
# 11-Nov-2009: Mario David (david@lip.pt). Removed CELL information from $jobID
#
# Description:
# Submission script for SGE, to be invoked by blahpd server.
# Usage:
# sge_submit.sh -c <command> [-i <stdin>] [-o <stdout>] [-e <stderr>] [-w working dir] [-- command's arguments]
#
# Copyright (c) Members of the EGEE Collaboration. 2004.
# See http://www.eu-egee.org/partners/ for details on the copyright
# holders.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#exec 2>> /tmp/submit.log
. /etc/lrms/ge_globals.conf
. `dirname $0`/blah_common_submit_functions.sh
if [ -z "$sge_rootpath" ]; then sge_rootpath="/usr/local/sge/pro"; fi
if [ -r "$sge_rootpath/${sge_cellname:-default}/common/settings.sh" ]
then
. $sge_rootpath/${sge_cellname:-default}/common/settings.sh
fi
bls_job_id_for_renewal=JOB_ID
original_args="$@"
bls_parse_submit_options "$@"
bls_setup_all_files
# Write wrapper preamble
cat > $bls_tmp_file << end_of_preamble
#!/bin/bash
# SGE job wrapper generated by `basename $0`
# on `/bin/date`
module load grid/egi
module load mpich2/1.3.2p1-gnu
#
# stgcmd = $bls_opt_stgcmd
# proxy_string = $bls_opt_proxy_string
# proxy_local_file = $bls_proxy_local_file
#
# SGE directives:
#\$ -S /bin/bash
end_of_preamble
#local batch system-specific file output must be added to the submit file
local_submit_attributes_file=${GLITE_LOCATION:-/opt/glite}/libexec/sge_local_submit_attributes.sh
if [ -r $local_submit_attributes_file ] ; then
echo \#\!/bin/sh > $bls_opt_tmp_req_file
if [ ! -z $bls_opt_req_file ] ; then
cat $bls_opt_req_file >> $bls_opt_tmp_req_file
fi
echo "source $local_submit_attributes_file" >> $bls_opt_tmp_req_file
chmod +x $bls_opt_tmp_req_file
$bls_opt_tmp_req_file >> $bls_tmp_file 2> /dev/null
rm -f $bls_opt_tmp_req_file
fi
if [ ! -z "$bls_opt_xtra_args" ] ; then
echo -e $bls_opt_xtra_args >> $bls_tmp_file 2> /dev/null
fi
# Write SGE directives according to command line options
# handle queue overriding
[ -z "$bls_opt_queue" ] || grep -q "^#\$ -q" $bls_tmp_file || echo "#\$ -q $bls_opt_queue" >> $bls_tmp_file
[ -z "$bls_opt_mpinodes" -o "x${bls_opt_mpinodes}" = "x1" ] || grep -q "^#\$ -pe *\\*" $bls_tmp_file || echo "#\$ -pe * $bls_opt_mpinodes" >>$bls_tmp_file
# Input and output sandbox setup.
bls_fl_subst_and_accumulate inputsand "@@F_REMOTE@`hostname -f`:@@F_LOCAL" "@@@"
[ -z "$bls_fl_subst_and_accumulate_result" ] || echo "#\$ -v SGE_stagein=$bls_fl_subst_and_accumulate_result" >> $bls_tmp_file
bls_fl_subst_and_accumulate outputsand "@@F_REMOTE@`hostname -f`:@@F_LOCAL" "@@@"
[ -z "$bls_fl_subst_and_accumulate_result" ] || echo "#\$ -v SGE_stageout=$bls_fl_subst_and_accumulate_result" >> $bls_tmp_file
echo "#$ -m n" >> $bls_tmp_file
# Write GE directives according to /etc/lrms/ge_local_submit_wrapper.conf
ge_retcode=`$GE_REQS_SCRIPT --user-attrs=${bls_opt_req_file} $bls_opt_queue $bls_tmp_file 2>&1`
[ '$ge_retcode' != '0' ] && echo "$ge_retcode" 1>&2
bls_add_job_wrapper
###############################################################
# Submit the script
###############################################################
#Your job 3236842 ("run") has been submitted
jobID=`qsub $bls_tmp_file 2> /dev/null | perl -ne 'print $1 if /^Your job (\d+) /;'` # actual submission
#jobID=`qsub -l s_rt=540000,s_vmem=4G,num_proc=1,h_stack=32M,arch=x86_64,processor=opteron_6174,h_fsize=10 $bls_tmp_file 2> /dev/null | perl -ne 'print $1 if /^Your job (\d+) /;'` # actual submission
retcode=$?
if [ "$retcode" != "0" -o -z "$jobID" ] ; then
rm -f $bls_tmp_file
exit 1
fi
# 11/11/09 Mario David fix (remove CELL)
#jobID=$jobID.${SGE_CELL:-default}
# Compose the blahp jobID ("sge/" + datenow + sge jobid)
# 11/11/09 Mario David fix
blahp_jobID=sge/`date +%Y%m%d%H%M%S`/$jobID
if [ "x$job_registry" != "x" ]; then
now=`date +%s`
let now=$now-1
${blah_sbin_directory}/blah_job_registry_add "$blahp_jobID" "$jobID" 1 $now "$bls_opt_creamjobid" "$bls_proxy_local_file" "$bls_opt_proxyrenew_numeric" "$bls_opt_proxy_subject"
fi
echo "BLAHP_JOBID_PREFIX$blahp_jobID"
bls_wrap_up_submit
exit $retcode The module lines at the beginning are to load the Grid Engine and MPI environment on the Worker Nodes.
Configuring MPI
- Creating a GE parallel environment
[root@cream ldif]# qconf -ap GRIDEGI_mpi
root@cream.egi.cesga.es added "GRIDEGI_mpi" to parallel environment list
[root@cream ldif]# qconf -mp GRIDEGI_mpi
pe_name GRIDEGI_mpi
slots 9999
user_lists NONE
xuser_lists NONE
start_proc_args /opt/cesga/sistemas/sge/tools/util/startmpi.sh -catch_rsh \
$pe_hostfile
stop_proc_args /opt/cesga/sistemas/sge/tools/util/stopmpi.sh
allocation_rule $pe_slots
control_slaves TRUE
job_is_first_task FALSE
urgency_slots min
accounting_summary FALSE- Configuring the parallel environment in the GRID queues:
[root@cream ldif]# qconf -sq GRIDEGI_large | grep -i pe_ pe_list GRIDEGI_mpi [root@cream ldif]# qconf -sq GRIDEGI_small | grep -i pe_ pe_list GRIDEGI_mpi [root@cream ldif]# qconf -sq GRIDEGI_ops | grep -i pe_ pe_list GRIDEGI_mpi
We added the following command on line 60 in the /usr/bin/sge_submit.sh script to load the right MPI environment and compilers. This is because we used the mpich installed on SVG cluster and we load the environment on the cluster using module application:
module load mpich2/1.3.2p1-gnu
BUGs/Known issues found after reconfiguring with YAIM
Problem at first configuration with EMI-2 CREAM with GE
The first yaim configuration for a EMI-2 CREAM CE using GE as batch system fails with: /etc/lrms/scheduler.conf: No such file or directory The problem disappears running again yaim
Problem if there are "extra" characters in the beginning of the tomcat key file
Because on an issue in trustmanager, there can be problems if there is something before -----BEGIN ...---- in /etc/grid-security/tomcat-key.pem The workaround is to simply remove these chars
Bug on the sge_submit.sh script
https://savannah.cern.ch/bugs/?95385
[root@cream grid-security]# diff /usr/libexec/sge_submit.sh /usr/libexec/sge_submit.sh_original
86c86
< [ -z "$bls_opt_mpinodes" -o "x${bls_opt_mpinodes}" = "x1" ] || grep -q "^#\$ -pe *\\*" $bls_tmp_file || echo "#\$ -pe * $bls_opt_mpinodes" >>$bls_tmp_file
---
> [ -z "$bls_opt_mpinodes" -o "x${bls_opt_mpinodes}" = "x1" ] || grep -q"^#\$ -pe *\\*" $bls_tmp_file || echo "#\$ -pe * $bls_opt_mpinodes" >>$bls_tmp_file
Removing sge ports from /etc/services
[root@cream ~]# grep 6444 /etc/services sge_qmaster 6444/tcp # Grid Engine Qmaster Service sge_qmaster 6444/udp # Grid Engine Qmaster Service [root@cream ~]# grep 6444 /etc/services ###sge_qmaster 6444/tcp # Grid Engine Qmaster Service ###sge_qmaster 6444/udp # Grid Engine Qmaster Service
It was necessary to load the SGE environment on the /var/lib/bdii/gip/plugin/glite-info-dynamic-ce script
[root@cream opssgm]# cat /var/lib/bdii/gip/plugin/glite-info-dynamic-ce #!/bin/sh . /etc/profile.d/sge.sh /usr/libexec/glite-ce-check-submission-state /etc/glite-ce-dbtool/creamdb_min_access.conf retcode=$? if [ $retcode = 1 ] || [ $retcode = 2 ] ; then # Submissions disabled. Publish Draining /usr/libexec/glite-info-dynamic-sge --info -c /etc/lrms/scheduler.conf | sed -e 's/GlueCEStateStatus: Production/GlueCEStateStatus: Draining/' else # Submissions enabled. The default cluster.state will be used /usr/libexec/glite-info-dynamic-sge --info -c /etc/lrms/scheduler.conf fi
Modifying /var/lib/bdii/gip/ldif/ComputingShare.ldif
It was necessary to edit /var/lib/bdii/gip/ldif/ComputingShare.ldif file and substitute the empty GLUE2PolicyRule attribute by GLUE2PolicyRule: VO:<name of the VO> :
Publishing the GlueCEPolicyMaxSlotsPerJob to run MPI jobs correctly
- Go to:
[root@cream ldif]# pwd /var/lib/bdii/gip/ldif
- And modify static-file-CE.ldif file to add on GlueCEPolicyMaxSlotsPerJob variable the maximum number of slots offered for MPI jobs:
[root@cream ldif]# grep -R GlueCEPolicyMaxSlotsPerJob ../ldif/* ../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8 ../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8 ../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8 [ ..... ]
If you obtain 999999, please substitute the static-file-CE.ldif by this one /root/static-file-CE.ldif_modificado
Increase the value for the memory in /usr/bin/apel-sge-log-parser
We had to change the following value in /usr/bin/apel-sge-log-parser file, because otherwise the APEL crond was not able to work. In our case, we have a big accounting file, therefore, the APEL crond needs pretty memory to be able to process the accounting file:
if [ -z $JAVA_HOME ]; then java -Xmx2524m -classpath $CP $LOG4J_PROPS org.glite.apel.sge.ApelLogParser "$@" else $JAVA_HOME/bin/java -Xmx2524m -classpath $CP $LOG4J_PROPS org.glite.apel.sge.ApelLogParser "$@" fi exit $?
The BUpdaterSGE daemons does not inherits the correct GE environment variables
Jobs were submitted without requesting complex due to this issue
If the environment present in a BUpdaterSGE process does not include the GE environment variables, the GE client commands (qstat, qconf) can not be executed by BUpdaterSGE. As a consequence, BUpdaterSGE will assume that jobs have been cancelled (because it receives no information from qstat or qacct). You can check the environment for BUpdaterSGE process using the following commands and searching for the GE env variables (SGE_EXECD, SGE_QMASTER, SGE_ROOT, SGE_CLUSTER_NAME, SGE_CELL) # ps xuawww | grep -i sge tomcat 7423 0.6 0.5 37184 21328 ? S Nov23 103:56 /usr/bin/BUpdaterSGE root 30622 0.0 0.0 61180 804 pts/0 R+ 13:41 0:00 grep -i sge # cat /proc/7423/environ This can happen if the BUpdaterSGE daemon is restarted by other user different than root (for example, tomcat starts the daemon at boot time and restarts it if the daemon is dead) without sourcing the proper environment. The workaround is to force the environment to be loaded in /etc/init.d/gLite and /etc/init.d/glite-ce-blahparser. This can be done simply by adding a line like the one bellow to be sourced at the beguinning of previous scripts . /etc/profile.d/sge.sh
Things to have in mind before reconfiguring with YAIM after some update
- Do backup of:
/etc/init.d/gLite /etc/init.d/glite-ce-blahparser
On these scripts must be loaded the SGE environment because otherwise jobs are submitted without requesting complex, this can be solved simply by adding a line like the one bellow to be sourced at the beginning of previous scripts
. /etc/profile.d/sge.sh
- Make backup of the following files
cd /var/lib/bdii/gip/plugin [root@cream plugin]# cp -aprx glite-info-dynamic-ce glite-info-dynamic-ce_mod
[root@cream plugin]# ll /usr/libexec/sge_submit.sh* -rwxr-xr-x 1 root root 4861 Feb 19 11:40 /usr/libexec/sge_submit.sh -rwxr-xr-x 1 root root 4861 Feb 19 11:40 /usr/libexec/sge_submit.sh_mod_OK
[root@cream libexec]# cp -aprx /var/lib/bdii/gip/ldif/ComputingShare.ldif /var/lib/bdii/gip/ldif/ComputingShare.ldif_mod_OK [root@cream libexec]#
- Publishing the GlueCEPolicyMaxSlotsPerJob to run MPI jobs correctly
- Go to:
[root@cream ldif]# pwd /var/lib/bdii/gip/ldif
- And check the you get the following output:
[root@cream ldif]# grep -R GlueCEPolicyMaxSlotsPerJob ../ldif/* ../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8 ../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8 ../ldif/static-file-CE.ldif:GlueCEPolicyMaxSlotsPerJob: 8 [ ..... ]
TESTING
Checking ldap (OK)
[root@ce2 ~]# ldapsearch -x -H ldap://cream.egi.cesga.es:2170 -b o=grid | grep 4444
[root@ce2 ~]# ldapsearch -x -H ldap://cream.egi.cesga.es:2170 -b o=grid | grep GlueCEStateWaitingJobs GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0 GlueCEStateWaitingJobs: 0
[root@ce2 ~]# ldapsearch -x -H ldap://cream.egi.cesga.es:2170 -b o=grid | grep GlueCEStateFreeCPUs GlueCEStateFreeCPUs: 90 GlueCEStateFreeCPUs: 27 GlueCEStateFreeCPUs: 58
[root@ce2 ~]# ldapsearch -x -H ldap://cream.egi.cesga.es:2170 -b o=grid | grep -i mpi [ .... ] GlueHostApplicationSoftwareRunTimeEnvironment: MPI-START GlueHostApplicationSoftwareRunTimeEnvironment: MPI-START-1.2.0 GlueHostApplicationSoftwareRunTimeEnvironment: MPICH2 GlueHostApplicationSoftwareRunTimeEnvironment: MPICH2-1.3.2p1 GlueHostApplicationSoftwareRunTimeEnvironment: MPI_SHARED_HOME
Submitting a job from the CESGA UI
[esfreire@ui ~]$ voms-proxy-init --voms ops:/ops/Role=lcgadmin Enter GRID pass phrase: Your identity: /DC=es/DC=irisgrid/O=cesga/CN=jesteban.freire Creating temporary proxy ............................ Done Contacting voms.cern.ch:15009 [/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch] "ops" Done Creating proxy ................................................................................... Done Your proxy is valid until Thu Feb 23 22:56:01 2012
[esfreire@ui ~]$ glite-ce-delegate-proxy -e cream.egi.cesga.es esfreire 2012-02-23 11:02:14,272 NOTICE - Proxy with delegation id [esfreire] succesfully delegated to endpoint [https://cream.egi.cesga.es:8443//ce-cream/services/gridsite-delegation] [esfreire@ui ~]$ glite-ce-proxy-renew -e cream.egi.cesga.es esfreire 2012-02-23 11:02:26,324 NOTICE - Proxy with delegation id [esfreire] succesfully renewed to endpoint [https://cream.egi.cesga.es:8443//ce-cream/services/gridsite-delegation]
[esfreire@ui ~]$ glite-ce-job-submit -D esfreire -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops lanzar_gromacs.jdl https://cream.egi.cesga.es:8443/CREAM421374529
[esfreire@ui ~]$ glite-ce-job-status https://cream.egi.cesga.es:8443/CREAM421374529
****** JobID=[https://cream.egi.cesga.es:8443/CREAM421374529]
Status = [DONE-OK]
ExitCode = [W][root@se2 tmp]# ll /tmp/ | grep -i gro -rw-r--r-- 1 opssgm004 opssgm 221898 Feb 23 11:05 job_gromacs_dir_output.tar.gz -rw-r--r-- 1 opssgm004 opssgm 103053 Feb 23 11:05 test_gromacs.err -rw-r--r-- 1 opssgm004 opssgm 44018 Feb 23 11:05 test_gromacs.out
- Using the gLite-WMS
[esfreire@ui ~]$ glite-wms-job-submit -a -o test_cream -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops lanzar_gromacs.jdl Connecting to the service https://wms01.egee.cesga.es:7443/glite_wms_wmproxy_server ====================== glite-wms-job-submit Success ====================== The job has been successfully submitted to the WMProxy Your job identifier is: https://wms01.egee.cesga.es:9000/yG5zkzKwYNwVUTkJa-RlVg The job identifier has been saved in the following file: /home/esfreire/test_cream ==========================================================================
[esfreire@ui ~]$ glite-wms-job-status -i test_cream
======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:
Status info for the Job : https://wms01.egee.cesga.es:9000/yG5zkzKwYNwVUTkJa-RlVg
Current Status: Done (Success)
Logged Reason(s):
- job completed
- Job Terminated Successfully
Exit code: 0
Status Reason: Job Terminated Successfully
Destination: cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops
Submitted: Thu Feb 23 16:22:32 2012 CET
==========================================================================esfreire@ui ~]$ glite-wms-job-output --dir jobOutput/ -i test_cream
Connecting to the service https://wms01.egee.cesga.es:7443/glite_wms_wmproxy_server
================================================================================
JOB GET OUTPUT OUTCOME
Output sandbox files for the job:
https://wms01.egee.cesga.es:9000/yG5zkzKwYNwVUTkJa-RlVg
have been successfully retrieved and stored in the directory:
/home/esfreire/jobOutput/esfreire_yG5zkzKwYNwVUTkJa-RlVg
================================================================================
Testing MPI jobs
[esfreire@ui mpi_test_CreamCE_prod]$ cat job1.jdl
CPUNumber = 4;
Executable = "/opt/exp_soft2/mpi-start/bin/mpi-start";
Arguments = "-v -pre hooks.sh cpi";
InputSandbox = {"cpi.c", "hooks.sh"};
StdOutput = "std.out";
StdError = "std.err";
OutputSandbox = {"std.out","std.err"};
##OutputSandboxBaseDestUri = "gsiftp://localhost";
OutputSandboxBaseDestUri = "gsiftp://se2.egi.cesga.es/tmp";
Requirements =
Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment) &&
Member("OPENMPI", other.GlueHostApplicationSoftwareRunTimeEnvironment);[esfreire@ui mpi_test_CreamCE_prod]$ cat hooks.sh
#!/bin/sh
pre_run_hook () {
# Compile the program.
echo "Compiling ${I2G_MPI_APPLICATION}"
sleep 20
# Actually compile the program.
cmd="mpicc ${MPI_MPICC_OPTS} -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c"
$cmd
if [ ! $? -eq 0 ]; then
echo "Error compiling program. Exiting..."
return 1
fi
# Everything's OK.
echo "Successfully compiled ${I2G_MPI_APPLICATION}"
return 0
}
Applying 4 CPUS (OK)
[esfreire@ui mpi_test_CreamCE_prod]$ glite-ce-job-submit -D esfreire -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops job1.jdl
https://cream.egi.cesga.es:8443/CREAM390858921
[esfreire@ui mpi_test_CreamCE_prod]$ glite-ce-job-status https://cream.egi.cesga.es:8443/CREAM390858921
****** JobID=[https://cream.egi.cesga.es:8443/CREAM390858921]
Status = [DONE-OK]
ExitCode = [0][root@se2 tmp]# cat std.err mpi-start [INFO ]: *********************************************************** mpi-start [INFO ]: UID = opssgm002 mpi-start [INFO ]: HOST = compute-5-1.local mpi-start [INFO ]: DATE = Tue Feb 28 10:17:48 CET 2012 mpi-start [INFO ]: VERSION = 1.3.0 mpi-start [INFO ]: *********************************************************** mpi-start [INFO ]: search for scheduler mpi-start [INFO ]: activate support for sge mpi-start [INFO ]: Detected 2 CPU socket(s) and 12 core(s) per CPU mpi-start [INFO ]: activate support for generic mpi-start [INFO ]: call backend MPI implementation mpi-start [INFO ]: start program with mpirun Process 0 on compute-5-1.local: n=0 Using 16384 intervals Process 1 on compute-5-1.local: n=0 Process 2 on compute-5-1.local: n=0 Process 3 on compute-5-1.local: n=0 [root@se2 tmp]# cat std.out Compiling cpi Successfully compiled cpi =[START]======================================================================= pi is approximately 3.1415926539002341, Error is 0.0000000003104410 wall clock time = 0.000208 =[FINISHED]====================================================================
Applying 8 CPUS (OK)
[root@se2 tmp]# cat std.err mpi-start [INFO ]: *********************************************************** mpi-start [INFO ]: UID = opssgm002 mpi-start [INFO ]: HOST = compute-5-1.local mpi-start [INFO ]: DATE = Tue Feb 28 10:29:10 CET 2012 mpi-start [INFO ]: VERSION = 1.3.0 mpi-start [INFO ]: *********************************************************** mpi-start [INFO ]: search for scheduler mpi-start [INFO ]: activate support for sge mpi-start [INFO ]: Detected 2 CPU socket(s) and 12 core(s) per CPU mpi-start [INFO ]: activate support for generic mpi-start [INFO ]: call backend MPI implementation mpi-start [INFO ]: start program with mpirun Process 0 on compute-5-1.local: n=0 Using 16384 intervals Process 1 on compute-5-1.local: n=0 Process 2 on compute-5-1.local: n=0 Process 3 on compute-5-1.local: n=0 Process 4 on compute-5-1.local: n=0 Process 5 on compute-5-1.local: n=0 Process 6 on compute-5-1.local: n=0 Process 7 on compute-5-1.local: n=0 [root@se2 tmp]# cat std.out Compiling cpi Successfully compiled cpi =[START]======================================================================= pi is approximately 3.1415926539002332, Error is 0.0000000003104401 wall clock time = 0.000416 =[FINISHED]====================================================================
Second MPI job - Applying 4 CPUS (OK)
[esfreire@ui mpi_test_CreamCE_prod]$ glite-ce-job-submit -D esfreire -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops job3.jdl https://cream.egi.cesga.es:8443/CREAM011898581 [root@se2 tmp]# cat std.err mpi-start [INFO ]: *********************************************************** mpi-start [INFO ]: UID = opssgm002 mpi-start [INFO ]: HOST = compute-5-1.local mpi-start [INFO ]: DATE = Tue Feb 28 10:35:23 CET 2012 mpi-start [INFO ]: VERSION = 1.3.0 mpi-start [INFO ]: *********************************************************** mpi-start [INFO ]: search for scheduler mpi-start [INFO ]: activate support for sge mpi-start [INFO ]: Detected 2 CPU socket(s) and 12 core(s) per CPU mpi-start [INFO ]: activate support for generic mpi-start [INFO ]: call backend MPI implementation mpi-start [INFO ]: start program with mpirun Process 2 on compute-5-1.local: n=0 Process 0 on compute-5-1.local: n=0 Using 16384 intervals Process 1 on compute-5-1.local: n=0 Process 3 on compute-5-1.local: n=0 [root@se2 tmp]# cat std.out Compiling cpi Successfully compiled cpi =[START]======================================================================= pi is approximately 3.1415926539002341, Error is 0.0000000003104410 wall clock time = 0.000544 =[FINISHED]====================================================================
jobOldSyntax - Applying 4 CPUS (OK)
glite-ce-job-submit -D esfreire -r cream.egi.cesga.es:8443/cream-sge-GRIDEGI_ops job-oldsyntax.jdl
[esfreire@ui mpi_test_CreamCE_prod]$ cat job-oldsyntax.jdl
CPUNumber = 4;
Executable = "starter.sh";
Arguments = "cpi";
InputSandbox = {"starter.sh", "cpi.c", "hooks.sh"};
StdOutput = "std.out";
StdError = "std.err";
OutputSandbox = {"std.out","std.err"};
OutputSandboxBaseDestUri = "gsiftp://se2.egi.cesga.es/tmp";
Environment = {"I2G_MPI_PRE_RUN_HOOK=hooks.sh"};
Requirements =
Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment) &&
Member("OPENMPI", other.GlueHostApplicationSoftwareRunTimeEnvironment);
[root@se2 tmp]# cat std.err
mpi-start [INFO ]: ***********************************************************
mpi-start [INFO ]: UID = opssgm002
mpi-start [INFO ]: HOST = compute-5-1.local
mpi-start [INFO ]: DATE = Tue Feb 28 10:43:11 CET 2012
mpi-start [INFO ]: VERSION = 1.3.0
mpi-start [INFO ]: ***********************************************************
mpi-start [INFO ]: search for scheduler
mpi-start [INFO ]: activate support for sge
mpi-start [INFO ]: Detected 2 CPU socket(s) and 12 core(s) per CPU
mpi-start [INFO ]: activate support for generic
mpi-start [INFO ]: call backend MPI implementation
mpi-start [INFO ]: start program with mpirun
Process 0 on compute-5-1.local: n=0
Using 16384 intervals
Process 1 on compute-5-1.local: n=0
Process 3 on compute-5-1.local: n=0
Process 2 on compute-5-1.local: n=0
[root@se2 tmp]# cat std.out
Compiling cpi
Successfully compiled cpi
=[START]=======================================================================
pi is approximately 3.1415926539002341, Error is 0.0000000003104410
wall clock time = 0.000533
=[FINISHED]====================================================================