DPM migration HOWTO

Contents

DPM migration HOWTO

Introduction

This guide describes the procedure to migrate an old DPM glite3.2 service to the latest EMI version stored at EGI UMD. We make some assumptions:

We will migrate our DPM system from machine A to B (EMI).
A and B have the same hostname and host certificate (DPM domain does not change.)
We will use UMD repository to install EMI middleware.
For this HOWTO we are using a DPM_mysql and DMP_disk installed in the same node. For different DPM_disk service we have to repeat the process.

Check the DPM version you were using (A)

We assume that you are trying to upgrade from the latest gLite 3.2 DPM version (1.8.2-3) [1]. If this is not the case, please have in mind that along the gLite 3.2 DPM lifecycle there were some DPM schema changes that may not longer be compatible with the latest UMD versions (a dump of the databases will not work).

Instructions on how to update to the latest schema are provided later on.

Backup old DPM (A)

This machine will be replaced by machine B. We have to backup our current DB and storage system first.

First steps (A)

Remember: First of all create a scheduled downtime for your DPM in GOCDB.
When scheduled downtime starts stop all DPM services except mysql. Services that you should stop (It they are running):
- dpm-httpd
- dpm-xrd
- dpm-manager-xrd
- dpm-manager-cms
- dpm-cms
- bdii
- srmv2.2
- srmv2
- srmv1
- dpmcopyd
- dpm
- rfiod
- dpm-gsiftp
- dpnsdaemon
Backup your storage system: If you are using a DMP_mysql+DPM_disk in the same machine probably /storage filesystem is mounted, you can unmount it or run a backup to store it in the new machine. Probably you have something like that in your /etc/fstab:

x.x.x.x:/<path>   /storage                nfs rw,rsize=32768,wsize=32768,bg,intr,noatime,tcp,nfsvers=3       0 0

Backup your DPM database (A)

In A you must dump and put in a safe place your DPM databases. This point is critical to maintain your DPM consistency. As root (look for root mysql pass in your site-info.def file):

mysqldump  --opt -u root -p'*****' cns_db > cns_db.sql
mysqldump  --opt -u root -p'*****' dpm_db > dpm_db.sql

Copy these files in a safe place (outside A).

Backup your yaim conf and host certs (A)

We will use the same conf and VOs for B so we can backup our yaim conf files: groups.conf, site-info.def, users.conf and vo.d

Copy your hosts certs in a safe place:

/etc/grid-security/hostcert.pem
/etc/grid-security/hostkey.pem

Copy these files in a safe place (outside A).
After that we can shut down our old machine. We can start with B installation.

New DPM-mysql installation (B)

Installing the UMD repository (B)

We will use an x86_64 SL5 OS to start. Preparing the system to install the UMD software, http://repository.egi.eu/category/umd_releases/distribution/umd_1/

   1. Remove all UMD and EPEL related configuration files that may exist from /etc/yum.repos.d so that there is no misconfigured epel or UMD repositories.
      Note be careful not to remove the OS or other necessary repo configuration files
      execute rm /etc/yum.repos.d/UMD* /etc/yum.repos.d/epel*
   2. Install epel-release-5-4.noarch.rpm
      execute wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
      yum install epel-release-5-4.noarch.rpm
   3. Install yum-priorities RPM
      execute yum install yum-priorities
   4. Install the umd-release RPM that adds all missing repositories that are needed for the installation of UMD products and configures yum priorities
      execute wget http://repository.egi.eu/sw/production/umd/1/sl5/x86_64/updates/umd-release-1.0.2-1.el5.noarch.rpm
      yum install umd-release-1.0.2-1.el5.noarch.rpm

The Production sites will need to configure the following repositories

    * The Main OS Repository
    * EGI Trust Anchor Repository: use the .repo files at EGI CAs repository, http://repository.egi.eu/sw/production/cas/1/current/repo-files/
    * UMD Production Repository: use the .repo files at UMD repository, http://repository.egi.eu/sw/production/umd/1/repofiles/

It is required to set the UMD repository with the highest priority in order to avoid possible discrepancies with EPEL.

In order to do this, you will need to install yum-priorities package and verify that the .repo file for UMD has

protect=1
priority=1

Also make sure that your /etc/yum/pluginconf.d/priorities.conf file has

[main]
enabled = 1
check_obsoletes = 1

Install DPM middleware (B)

Install the new middleware (In this case we will install a dpm-mysql head node and dpm-disk in the same node). Run:

yum clean all
yum install emi-dpm_mysql emi-dpm_disk

Mount or copy dpm storage system in the new machine. If you storage system is shared by nfs you have to add the mount point again in B. Check your /etc/fstab file:

x.x.x.x:/<path>   /storage                nfs rw,rsize=32768,wsize=32768,bg,intr,noatime,tcp,nfsvers=3       0 0

$/!\$ IMPORTANT: Remember, you must mount the storage system in the same path!
Copy yaim conf backup files to B and run yaim.

/opt/glite/yaim/bin/yaim -c -s site-info.def  -n emi_dpm_mysql -n emi_dpm_disk

At this point you have a new DPM service using UMD repos. You must recover your old database to find again your files.

Recover DPM information (B)

We will recover all DPM info in our new DPM service, like in A we have to stop all DPM services except mysql daemon.

Copy mysql dumps to B and recover the database running this command. (This will take some time... time for a coffee break )

mysql -u root -p'******' cns_db < cns_db.sql
mysql -u root -p'*******' dpm_db < dpm_db.sql

* If you have used a database dump from a DPM version <= 1.8.1 in your new UMD/EMI service, this will not work. The workaround is to do the upgrade by hand, by running a script against the database manually [2,3,4]. In summary, the changes are quoted hereafter (please check if all of them apply to your case). Version 1.8.4 will have the fix for this issue

USE dpm_db
ALTER TABLE dpm_fs ADD weight INTEGER;
UPDATE  dpm_fs SET weight=1;
UPDATE schema_version_dpm SET major = 3, minor = 3, patch = 0;
 
USE cns_db
ALTER TABLE Cns_userinfo ADD user_ca VARCHAR(255) BINARY,
ALTER TABLE Cns_userinfo banned INTEGER;
UPDATE  Cns_userinfo SET banned=0;
ALTER TABLE Cns_groupinfo banned INTEGER;
UPDATE  Cns_groupinfo SET banned=0;
UPDATE schema_version SET major = 3, minor = 1, patch = 0;

Check if your old database info is there:

# mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 260
Server version: 5.0.77 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema | 
| cns_db             | 
| dpm_db             | 
| mysql              | 
| test               | 
+--------------------+
5 rows in set (0.00 sec)

mysql> use dpm_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+--------------------+
| Tables_in_dpm_db   |
+--------------------+
| dpm_copy_filereq   | 
| dpm_fs             | 
| dpm_get_filereq    | 
| dpm_pending_req    | 
| dpm_pool           | 
| dpm_put_filereq    | 
| dpm_req            | 
| dpm_space_reserv   | 
| dpm_unique_id      | 
| schema_version_dpm | 
+--------------------+
10 rows in set (0.00 sec)
mysql> select * from dpm_get_filereq limit 5;

Check also VO directories permissions (should belong to dpmmgr user and group). In our case we are using /storage dir.

# cd /storage/
# ls -l
total 416
drwxrwxr-x  132 dpmmgr dpmmgr  7168 Mar 14  2011 alice
drwxrwxr-x  589 dpmmgr dpmmgr 28672 Dec 26 07:14 atlas
...
...

You will see your old data and srm endpoints, If so good news ! your system is restored. Now for clean up some things we can re-run yaim conf. (It will check our database schema and restart dpm services).

/opt/glite/yaim/bin/yaim -c -s site-info.def  -n emi_dpm_mysql -n emi_dpm_disk

Known Issues and final checks (B)

It exists a monthly cron to create new VO directories for each month, this script should be executed by hand the first time (/etc/cron.monthly/create-default-dirs-DPM.sh)

$/!\$ KNOWN BUG: /etc/cron.monthly/create-default-dirs-DPM.sh script does not have LCG_LOCATION variable set and it will fail.

This is a known bug and it will fixed in the next UMD1.5 release. The workaround is quite simple we have to add this line into it:

...
LCG_LOCATION="/usr"
...

Now we can run this script as root:

/etc/cron.monthly/create-default-dirs-DPM.sh

We have to wait for nagios tests, if everything goes smooth we can remove the scheduled downtime and use our new DPM!

DPM_migration_gLite3.2_to_UMD