How to prevent Cream Direct Job Submissions from Timing out

Problem / issue

• Cream Direct Job Submissions reach the 2 minutes timeout although glite-ce-cream.log detect that the job finished with DONE-OK, and glite-ce-bnotifier.log acknowledges the JobStatus=4 (completed).

Possible source

• This could be due to performance problems at the CREAM DB level, for example, as the ones bellow in glite-ce-cream.log

07 Sep 2012 15:22:48,467 ERROR org.glite.ce.cream.cmdmanagement.queue.db.CommandQueueDBManager (CommandQueueDBManager.java:722) - (pool-1-thread-2) Operation failed: Lock wait timeout exceeded; try restarting transaction
07 Sep 2012 15:22:48,469 ERROR org.glite.ce.cream.cmdmanagement.CommandManager (CommandManager.java:356) - (pool-1-thread-2) cannot enqueue the command SET_JOB_STATUS: Lock wait timeout exceeded; try restarting transaction
07 Sep 2012 15:22:48,477 ERROR org.glite.ce.cream.jobmanagement.cmdexecutor.blah.LRMSEventsProcessor (LRMSEventsProcessor.java:178) - (pool-1-thread-2) cannot enqueue the command SET_JOB_STATUS: Lock wait timeout exceeded; try restarting transaction
07 Sep 2012 15:22:49,469 ERROR org.glite.ce.cream.cmdmanagement.queue.db.CommandQueueDBManager (CommandQueueDBManager.java:722) - (pool-1-thread-4) Operation failed: Lock wait timeout exceeded; try restarting transaction

Possible solution

• Increase innodb_lock_wait_timeout (default is 50, now 120) in my.cnf and restart mysql