Pos-EMI GE Plan

Review Log

• Version 1.2 (Date: 21th May 2013)
  • Create CreamCE Setup wiki page (following Esteban notes)
  • Update Issues Follow up table
  • Information for pos-EMI releases
  • Action update
• Version 1.1 (Date: 18th April 2013)
  • Proposal for code collaboration with BLAH
  • Instructions for BLAH compilation standalone
  • Action update
• Version 1.0 (Date: 17th April 2013)
  • First Draft

Handover Information

EMI will finish in April 2013. With the end of the project, Roberto Rosende (leading GE utils CreamCE integration) will leave. CESGA will continue the support of the GE_utils integration with CREAM but not at 100%. Roberto should handover current work either to Ivan Diaz or Alvaro Simon. IFCA will continue to support the information provider, and LIP will continue with yaim (or other configuration tool that might be adopted in the future)

Mailing Lists and Contact Points

Contact points for the CREAM integration:

1. paolo.andreetto@pd.infn.it : CREAM / BLAH Integration Team

2. blah-cream@lists.infn.it : BLAH issues and plans. The responsible people assuming Roberto's tasks should subscribe

3. emt@eu-emi.eu , all-members@eu-emi.eu : General EMI mailing lists may also be interesting in order to understand the future of all software products

Issues Follow-up

Status Report:

1. The first EMI-3 release (which is also supposed to be released as an UMD-2 update) will fix issue #1 and #4

2. Issue #2 will not be fixed in the terms exposed by the submitters (IN2P3). An alternative mechanism to add local variables with a hook was implemented but it does not fit the submitters. We proposed to close it as unsolved.

3. The alternative mechanism to add local variables was not released but it should, since it may be useful to many other sites. It was developed at IFCA and it is in use at IFCA and CESGA. A new ticket (issue #5) has been opened in Trac@IFCA to follow this up. Esteban is supposed to send documentation prepared @CESGA on how they actually implemented it.
4. Issues #3 and 6 are not really addressed.

Source Code

Repositories

The code is currently dispersed in the following repositories:

1. Information Provider: https://github.com/italiangrid/info-dynamic-scheduler-ge

2. yaim: https://github.com/grid-admin/GE-utils

3. blah: https://github.com/prelz/BLAH/ (Old gLite repository: http://glite.cvs.cern.ch/cgi-bin/glite.cgi/org.glite.ce.blahp)

Source Code Collaboration proposal

We should concentrate the code on github as follows:

1. https://github.com/ge-utils/blah (a fork from BLAH repo)

2. https://github.com/ge-utils/yaim-functions

3. https://github.com/ge-utils/info-dynamic-scheduler

For this to happen, we should:

1. Guarantee that the the SVN @ IFCA is synchronized with the software currently delivered by the EMI-3 release.
2. Migrate the existing SVN repos (yaim, infomation provider) to their ge-utils git equivalents

3. Clone the original BLAH repo to its ge-utils git equivalent

The proposed model to contribute to BLAH is the following:

1. We fork the BLAH repo (which should contain the most recent GE_Utils BLAH code) to https://github.com/ge-utils/blah.

2. In https://github.com/ge-utils/blah we keep the master branch in a production-like state (i.e.code that is ready to deployed and ready to be submitted for release. it does not need to be aligned with BLAH at all times).

3. We make the developments in topic branches. Once the development is done, we can think in merging it with the master branch.

4. BEFORE merging that topic branch into our master branch, we have to sync the master branch with the BLAH repo. This way, we can then submit a pull request for our changes.

###
### Init the repo. These two first steps should be made only once:
###

# clone OUR repo
git clone https://github.com/ge-utils/blah

# add their remote, so that we can get changes from upstream
# This is really important to guarantee that we keep our code synchronized
git remote add upstream https://github.com/prelz/BLAH/

###
### Working in a new feature
###

# 1st get the latest changes
git pull upstream master

# create a topic branch starting from the master branch
git checkout -b brand_new_feature master

# hack the code, submit the changes
git commit -a

# push the changes to the "brand_new_feature" of ge-utils/blah
git push origin brand_new_feature


###
### After some more commits, we are ready for merge. Make a self pull request
###

# update the master branch with upstream changes
git checkout master
git pull upstream master
git push origin master

# go to the github page, make a pull request, or use the CLI hub
hub pull-request -h ge-utils/blah:brand_new_feature hub pull-request -h ge-utils/blah:master

# Do we need some more changes? Simply commit and push the branch again.
# The pull request will be updated automatically. Just make sure that we are in the correct branch:
git branch
git checkout brand_new_feature
git commit -a
git push origin brand_new_feature


###
### Now the changes are in github and in OUR master branch. We can start a pull
### request right now from the web, or use the "hub" CLI to do so:
hub pull-request -h ge-utils/blah:master -b https://github.com/prelz/BLAH/:master

Standalone BLAH compilation

###
### This compilation was done on a SL6 machine (EPEL repo should be enabled)
###

# Install necessary software
$ yum install gcc gcc-c++ cmake git rpm-build classads-devel docbook-xsl globus-gss-assist-devel globus-gsi-credential-devel globus-gsi-proxy-core-devel globus-gsi-cert-utils-devel globus-io-devel

# Clone the repository
$ mkdir BLAH_COMPILATION; cd BLAH_COMPILATION; git clone https://github.com/prelz/BLAH/

# Compile
$ cd BLAH; cmake .

# Produce the rpm
$ make BLAH_rpm

# rpm -qpl RPM/RPMS/x86_64/BLAH-1.21.0-1.x86_64.rpm
/etc/blah.config.template
/etc/blparser.conf.template
/etc/rc.d/init.d
/etc/rc.d/init.d/glite-ce-blah-parser
/usr/bin/blahpd
/usr/libexec/BLClient
/usr/libexec/BLParserLSF
/usr/libexec/BLParserPBS
/usr/libexec/BNotifier
/usr/libexec/BPRclient
/usr/libexec/BPRserver
/usr/libexec/BUpdaterCondor
/usr/libexec/BUpdaterLSF
/usr/libexec/BUpdaterPBS
/usr/libexec/BUpdaterSGE
/usr/libexec/BUpdaterSLURM
/usr/libexec/blah_common_submit_functions.sh
/usr/libexec/blah_load_config.sh
/usr/libexec/blparser_master
/usr/libexec/condor_cancel.sh
/usr/libexec/condor_hold.sh
/usr/libexec/condor_resume.sh
/usr/libexec/condor_status.sh
/usr/libexec/condor_submit.sh
/usr/libexec/lsf_cancel.sh
/usr/libexec/lsf_hold.sh
/usr/libexec/lsf_resume.sh
/usr/libexec/lsf_status.sh
/usr/libexec/lsf_submit.sh
/usr/libexec/pbs_cancel.sh
/usr/libexec/pbs_hold.sh
/usr/libexec/pbs_resume.sh
/usr/libexec/pbs_status.sh
/usr/libexec/pbs_submit.sh
/usr/libexec/runcmd.pl.template
/usr/libexec/sge_cancel.sh
/usr/libexec/sge_filestaging
/usr/libexec/sge_helper
/usr/libexec/sge_hold.sh
/usr/libexec/sge_local_submit_attributes.sh
/usr/libexec/sge_resume.sh
/usr/libexec/sge_status.sh
/usr/libexec/sge_submit.sh
/usr/libexec/slurm_cancel.sh
/usr/libexec/slurm_hold.sh
/usr/libexec/slurm_resume.sh
/usr/libexec/slurm_status.sh
/usr/libexec/slurm_submit.sh
/usr/man/man1/blah_check_config.1.gz
/usr/man/man1/blah_job_registry_add.1.gz
/usr/man/man1/blah_job_registry_dump.1.gz
/usr/man/man1/blah_job_registry_lkup.1.gz
/usr/man/man1/blah_job_registry_scan_by_subject.1.gz
/usr/man/man1/blahpd.1.gz
/usr/man/man1/blahpd_daemon.1.gz
/usr/sbin/blah_check_config
/usr/sbin/blah_job_registry_add
/usr/sbin/blah_job_registry_dump
/usr/sbin/blah_job_registry_lkup
/usr/sbin/blah_job_registry_purge
/usr/sbin/blah_job_registry_scan_by_subject
/usr/sbin/blahpd_daemon
/usr/sbin/glite-ce-check-blparser
/usr/share/doc/BLAH-1.21.0
/usr/share/doc/BLAH-1.21.0/LICENSE

Information for pos-EMI releases

Until April 2014 all PTs agreed to maintain and support EMI versions with different "level" of support (as for example best-effort). PTs must:

• Prepare and test the packages

• Make them available in a public space (like GitHub)

• prepare Release Notes for the version you intend to release (like GitHub)

Cristina Aiftimiei ( cristina.aiftimiei@pd.infn.it ) agreed to collect updated communications from the PTs, announced via EMI EMT mailing list. PTs should provide the details above, providing links were the information is available. Cristina will then consolidate this information once a month, create a common announcement and put the information provided in the EMI pages.

Actions

References

[1] http://ibergrid.lip.pt/GE/CreamCESetUp#Assigning_complex_to_the_jobs_and_modifying_the_sge_submit.sh_script_for_it