Locked History Actions

GE/PlanAfterEMI

Pos-EMI GE Plan

Review Log

  • Version 1.2 (Date: 21th May 2013)
    • Create CreamCE Setup wiki page (following Esteban notes)
    • Update Issues Follow up table
    • Information for pos-EMI releases
    • Action update
  • Version 1.1 (Date: 18th April 2013)
    • Proposal for code collaboration with BLAH
    • Instructions for BLAH compilation standalone
    • Action update
  • Version 1.0 (Date: 17th April 2013)
    • First Draft

Handover Information

EMI will finish in April 2013. With the end of the project, Roberto Rosende (leading GE utils CreamCE integration) will leave. CESGA will continue the support of the GE_utils integration with CREAM but not at 100%. Roberto should handover current work either to Ivan Diaz or Alvaro Simon. IFCA will continue to support the information provider, and LIP will continue with yaim (or other configuration tool that might be adopted in the future)

Mailing Lists and Contact Points

Contact points for the CREAM integration:

  1. paolo.andreetto@pd.infn.it : CREAM / BLAH Integration Team

  2. blah-cream@lists.infn.it : BLAH issues and plans. The responsible people assuming Roberto's tasks should subscribe

  3. emt@eu-emi.eu , all-members@eu-emi.eu : General EMI mailing lists may also be interesting in order to understand the future of all software products

Issues Follow-up

Status Report:

  1. The first EMI-3 release (which is also supposed to be released as an UMD-2 update) will fix issue #1 and #4

  2. Issue #2 will not be fixed in the terms exposed by the submitters (IN2P3). An alternative mechanism to add local variables with a hook was implemented but it does not fit the submitters. We proposed to close it as unsolved.

  3. The alternative mechanism to add local variables was not released but it should, since it may be useful to many other sites. It was developed at IFCA and it is in use at IFCA and CESGA. A new ticket (issue #5) has been opened in Trac@IFCA to follow this up. Esteban is supposed to send documentation prepared @CESGA on how they actually implemented it.
  4. Issues #3 and 6 are not really addressed.


Issue

Summary

Domain

GGUS ID

Bug Tracker ID

Status

EMI-2 Task

EMI-3 Task

Notes

1

Info provider needs a cache mechanism for qstat data

Information Provider

#78607

Trac@IFCA #23

Fixed

EMI-2: #38301 (Ready for Testbed)

EMI-3: #34041 (Released)

2

Expand the CREAM-GE plug-in functionalities.

BLAH

#78955

Savannah@CERN #91612

Will not fix

NA

NA

3

Missing real value for GlueCEPolicyMaxSlotsPerJob

Information Provider

#82902

Savannah@CERN #95183

Will Fix

4

EMI2 SL6 CREAM-CE (SGE)/BDII failed in some published values

BLAH

#83352;#90353

Savannah@CERN #95586

Fixed

EMI-2: #38301 (Ready for Testbed)

EMI-3: #34041 (Released)

5

ge_reqs - Road to Production

BLAH

Trac@IFCA #24

Will Fix

[1]

6

Support for multi-core envs in GE

BLAH

Savannah@CERN #78332

Will Fix

7

Information Provider SegFault

Information Provider

GGUS #93506

Fixed in GitHub (Master, branch: glite-info-dynamic-ge-6.0.0.5)

8

Verification: CREAM-GE does not cancel jobs

BLAH

GGUS #94128

Fixed in GE BLAH GitHub (Master)

Already pushed to official BLAH GitHub. Will be delivered in EMI3 Update 6

9

sge_local_submit_attributes.sh memory limits

BLAH

GGUS #93956

Fixed in GE BLAH GitHub (Master)

Already pushed to official BLAH GitHub. Will be delivered in EMI3 Update 6

10

Verification: ge utils yaim configuration error for son of ge version 8.1.3

YAIM

GGUS #93920

Fixed in GE YAIM GitHub (Master, branch: glite-yaim-ge-utils-4.3.0-6).

Will be delivered in EMI3 Update 5

11

yaim fails to create the file /etc/lrms/scheduler.conf in a EMI2/UMD2 scratch installation

Information Provider

GGUS #94312

Fixed by CREAM PT

Will be delivered in EMI3 Update 6

12

Env var are not passed to BLAH daemons

BLAH

GGUS #88284

Fixed by BLAH PT

Will be delivered in EMI3 Update 6. check if yaim SGE will not destroy it

13

Configurable sleep times for BUpdater between multiple qacct commands

BLAH

Source Code

Repositories

The code is currently dispersed in the following repositories:

  1. Information Provider: https://github.com/italiangrid/info-dynamic-scheduler-ge

  2. yaim: https://github.com/grid-admin/GE-utils

  3. blah: https://github.com/prelz/BLAH/ (Old gLite repository: http://glite.cvs.cern.ch/cgi-bin/glite.cgi/org.glite.ce.blahp)

Source Code Collaboration proposal

We should concentrate the code on github as follows:

  1. https://github.com/ge-utils/blah (a fork from BLAH repo)

  2. https://github.com/ge-utils/yaim-functions

  3. https://github.com/ge-utils/info-dynamic-scheduler

For this to happen, we should:

  1. Guarantee that the the SVN @ IFCA is synchronized with the software currently delivered by the EMI-3 release.
  2. Migrate the existing SVN repos (yaim, infomation provider) to their ge-utils git equivalents
  3. Clone the original BLAH repo to its ge-utils git equivalent

The proposed model to contribute to BLAH is the following:

  1. We fork the BLAH repo (which should contain the most recent GE_Utils BLAH code) to https://github.com/ge-utils/blah.

  2. In https://github.com/ge-utils/blah we keep the master branch in a production-like state (i.e.code that is ready to deployed and ready to be submitted for release. it does not need to be aligned with BLAH at all times).

  3. We make the developments in topic branches. Once the development is done, we can think in merging it with the master branch.
  4. BEFORE merging that topic branch into our master branch, we have to sync the master branch with the BLAH repo. This way, we can then submit a pull request for our changes.

###
### Init the repo. These two first steps should be made only once:
###

# clone OUR repo
git clone https://github.com/ge-utils/blah

# add their remote, so that we can get changes from upstream
# This is really important to guarantee that we keep our code synchronized
git remote add upstream https://github.com/prelz/BLAH/

###
### Working in a new feature
###

# 1st get the latest changes
git pull upstream master

# create a topic branch starting from the master branch
git checkout -b brand_new_feature master

# hack the code, submit the changes
git commit -a

# push the changes to the "brand_new_feature" of ge-utils/blah
git push origin brand_new_feature


###
### After some more commits, we are ready for merge. Make a self pull request
###

# update the master branch with upstream changes
git checkout master
git pull upstream master
git push origin master

# go to the github page, make a pull request, or use the CLI hub
hub pull-request -h ge-utils/blah:brand_new_feature hub pull-request -h ge-utils/blah:master

# Do we need some more changes? Simply commit and push the branch again.
# The pull request will be updated automatically. Just make sure that we are in the correct branch:
git branch
git checkout brand_new_feature
git commit -a
git push origin brand_new_feature


###
### Now the changes are in github and in OUR master branch. We can start a pull
### request right now from the web, or use the "hub" CLI to do so:
hub pull-request -h ge-utils/blah:master -b https://github.com/prelz/BLAH/:master

Standalone BLAH compilation

###
### This compilation was done on a SL6 machine (EPEL repo should be enabled)
###

# Install necessary software
$ yum install gcc gcc-c++ cmake git rpm-build classads-devel docbook-xsl globus-gss-assist-devel globus-gsi-credential-devel globus-gsi-proxy-core-devel globus-gsi-cert-utils-devel globus-io-devel

# Clone the repository
$ mkdir BLAH_COMPILATION; cd BLAH_COMPILATION; git clone https://github.com/prelz/BLAH/

# Compile
$ cd BLAH; cmake .

# Produce the rpm
$ make BLAH_rpm

# rpm -qpl RPM/RPMS/x86_64/BLAH-1.21.0-1.x86_64.rpm
/etc/blah.config.template
/etc/blparser.conf.template
/etc/rc.d/init.d
/etc/rc.d/init.d/glite-ce-blah-parser
/usr/bin/blahpd
/usr/libexec/BLClient
/usr/libexec/BLParserLSF
/usr/libexec/BLParserPBS
/usr/libexec/BNotifier
/usr/libexec/BPRclient
/usr/libexec/BPRserver
/usr/libexec/BUpdaterCondor
/usr/libexec/BUpdaterLSF
/usr/libexec/BUpdaterPBS
/usr/libexec/BUpdaterSGE
/usr/libexec/BUpdaterSLURM
/usr/libexec/blah_common_submit_functions.sh
/usr/libexec/blah_load_config.sh
/usr/libexec/blparser_master
/usr/libexec/condor_cancel.sh
/usr/libexec/condor_hold.sh
/usr/libexec/condor_resume.sh
/usr/libexec/condor_status.sh
/usr/libexec/condor_submit.sh
/usr/libexec/lsf_cancel.sh
/usr/libexec/lsf_hold.sh
/usr/libexec/lsf_resume.sh
/usr/libexec/lsf_status.sh
/usr/libexec/lsf_submit.sh
/usr/libexec/pbs_cancel.sh
/usr/libexec/pbs_hold.sh
/usr/libexec/pbs_resume.sh
/usr/libexec/pbs_status.sh
/usr/libexec/pbs_submit.sh
/usr/libexec/runcmd.pl.template
/usr/libexec/sge_cancel.sh
/usr/libexec/sge_filestaging
/usr/libexec/sge_helper
/usr/libexec/sge_hold.sh
/usr/libexec/sge_local_submit_attributes.sh
/usr/libexec/sge_resume.sh
/usr/libexec/sge_status.sh
/usr/libexec/sge_submit.sh
/usr/libexec/slurm_cancel.sh
/usr/libexec/slurm_hold.sh
/usr/libexec/slurm_resume.sh
/usr/libexec/slurm_status.sh
/usr/libexec/slurm_submit.sh
/usr/man/man1/blah_check_config.1.gz
/usr/man/man1/blah_job_registry_add.1.gz
/usr/man/man1/blah_job_registry_dump.1.gz
/usr/man/man1/blah_job_registry_lkup.1.gz
/usr/man/man1/blah_job_registry_scan_by_subject.1.gz
/usr/man/man1/blahpd.1.gz
/usr/man/man1/blahpd_daemon.1.gz
/usr/sbin/blah_check_config
/usr/sbin/blah_job_registry_add
/usr/sbin/blah_job_registry_dump
/usr/sbin/blah_job_registry_lkup
/usr/sbin/blah_job_registry_purge
/usr/sbin/blah_job_registry_scan_by_subject
/usr/sbin/blahpd_daemon
/usr/sbin/glite-ce-check-blparser
/usr/share/doc/BLAH-1.21.0
/usr/share/doc/BLAH-1.21.0/LICENSE

Information for pos-EMI releases

Until April 2014 all PTs agreed to maintain and support EMI versions with different "level" of support (as for example best-effort). PTs must:

  • Prepare and test the packages
  • Make them available in a public space (like GitHub)

  • prepare Release Notes for the version you intend to release (like GitHub)

Cristina Aiftimiei ( cristina.aiftimiei@pd.infn.it ) agreed to collect updated communications from the PTs, announced via EMI EMT mailing list. PTs should provide the details above, providing links were the information is available. Cristina will then consolidate this information once a month, create a common announcement and put the information provided in the EMI pages.

Actions

ID

Action

Assigned To

Status

1

Update all GGUS and Savannah Tickets

R. Rosende

Open

2

Join to emt and blah-cream mailing lists

I.Diaz / E. Freire /A. Simon / R. Rosende

Open

3

Send documentation regarding implementation of local variables using ge_reqs

E. Freire

Done

4

Review current GE code with Roberto

I.Diaz / E. Freire /A. Simon / R. Rosende

Done

5

Guarantee that the production software (yaim and ge information provider) is sync with the SVN @ IFCA

R. Rosende

Done

6

Migrate the SVN code (yaim) to GITHUB (https://github.com/ge-utils/yaim-functions)

R. Rosende

Done

7

Migrate the SVN code (ge info provider) to GITHUB (https://github.com/ge-utils/info-dynamic-scheduler)

R. Rosende

Done

8

Fork original BLAH repository to GITHUB (https://github.com/ge-utils/blah)

A. López

Done

9

Provide authorization to all members in all github repos

A. López / R. Rosende

Done

9

Provide feedback regarding the future of the middleware deployment process (after Rome MEDIA meeting)

E. Fernandez

Done

10

Contact with BLAH staff and understand how should we continue to collaborate

G. Borges

Done

11

Explore other configuration utilities besides YAIM

G. Borges

Open

References

[1] http://ibergrid.lip.pt/GE/CreamCESetUp#Assigning_complex_to_the_jobs_and_modifying_the_sge_submit.sh_script_for_it