Changes between Initial Version and Version 1 of InstallationGuide

Timestamp:

05/27/13 23:14:50 (11 years ago)

Author:

mmamonski

Comment:

--

InstallationGuide

@@ +1 @@
+[[PageOutline]] 
+
+= Introduction =
+QCG-Computing service (the successor of the OpenDSP project) is an open source service acting as a computing provider exposing on demand access to computing resources and jobs over the HPC Basic Profile compliant Web Services interface. In addition the QCG-Computing offers remote interface for Advance Reservations management. 
+
+This document describes installation of the QCG-Computing service on Debian machines using binary packages. The service should be deployed on the machine (or virtual machine) that:
+* has at least 1GB of memory (recommended value: 2 GB)
+* has 10 GB of free disk space (most of the space will be used by the log files)
+* has any modern CPU (if you plan to use virtual machine you should dedicated to it one or two cores from the host machine)
+* runs DEBIAN 6.X
+= Prerequisites =
+We assume that you have the local batch systems  already installed.
+
+The !QosCosGrid services do not require from you to install any QCG component on the worker nodes, however application wrapper scripts need the following software to be available on worker nodes:
+ * bash, 
+ * rsync,
+ * zip/unzip,
+ * dos2unix,
+ * nc,
+ * python.
+Which are usually available out of the box on most of the HPC systems.
+
+== GridFTP server ==
+To be fully operable the !QosCosGrid stack requires the GridFTP server to be installed. This requirements is usually fulfilled by most PRACE sites. If not it can be easily installed by issuing the following commands:
+{{{
+# apt-get install xinetd globus-gridftp-server-progs 
+# cat > /etc/xinetd.d/gsiftp << EOF
+service gsiftp
+{
+ instances               = 100
+ socket_type             = stream
+ wait                    = no
+ user                    = root
+ env                     += GLOBUS_TCP_PORT_RANGE=20000,25000
+ server = /usr/sbin/globus-gridftp-server
+ server_args = -i -aa -l /var/log/globus-gridftp.log
+ server_args += -d WARN
+ log_on_success          += DURATION
+ nice                    = 10
+ disable                 = no
+}
+EOF
+# /etc/init.d/xinetd reload
+Reloading internet superserver configuration: xinetd.
+}}}
+= Firewall configuration =
+In order to expose the !QosCosGrid services externally you need to open the following incoming ports in the firewall:
+* 19000 (TCP) - QCG-Computing
+* 19001 (TCP) - QCG-Notification
+* 2811 (TCP) - GridFTP server
+* 20000-25000 (TCP) - GridFTP  port-range 
+
+The following outgoing trafic should be allowed in general:
+* NTP, DNS, HTTP, HTTPS services
+* gridftp (TCP ports: 2811 and port-ranges: 9000-9500, 20000-25000)
+
+= Related software =
+* Install database backend (PostgresSQL) - optional, only if you want to host the QCG-Computing database on the same machine. 
+{{{#!sh
+apt-get install postgresql
+}}}
+
+* UnixODBC and the PostgresSQL ODBC driver:
+{{{#!sh
+apt-get install unixodbc odbc-postgresql
+}}}
+
+Moreover we further assume that the X.509 host certificate and key are already installed in the following locations:
+* `/etc/grid-security/hostcert.pem`
+* `/etc/grid-security/hostkey.pem`
+
+Most of the grid services and security infrastructures are sensitive to time skews. Thus we recommended to install a Network Time Protocol daemon or use any other solution that provides accurate clock synchronization.
+
+= Installation  =
+The one who want to install QCG-Computing on Debian should follow these steps:
+
+* ensure that the qcg-comp user is present in a system, otherwise create it:
+{{{#!sh
+useradd -r -d /var/log/qcg-comp/ qcg-comp
+}}}
+
+* ensure that the qcg-dev group is present in a system, otherwise create it:
+{{{#!sh
+groupadd -r qcg-dev
+}}}
+
+* install the !QosCosGrid Debian repository:
+{{{#!sh
+cat > /etc/apt/sources.list.d/qcg.unstable.list << EOF
+deb http://fury.man.poznan.pl/qcg-packages/debian/ unstable main
+EOF
+}}}
+
+* add the public key of the QCG repository to your trusted keys in the apt configuration:
+{{{#!sh
+wget https://apps.man.poznan.pl/trac/qcg-notification/raw-attachment/wiki/InstallingUsingDeb/qcg.pub
+apt-key add qcg.pub
+}}}
+
+* refresh the packages list:
+{{{#!sh
+apt-get update
+}}}
+
+* install QCG-Computing:
+{{{
+#!div style="font-size: 90%"
+{{{#!sh
+apt-get install qcg-comp qcg-comp-client qcg-comp-doc
+}}}
+}}}
+
+* setup the QCG-Computing database as described [http://apps.man.poznan.pl/trac/qcg-computing/wiki/InstallingFromSources#Databasesetup here].
+
+
+
+
+= Service certificates =
+Copy the service certificate and key into the `/etc/qcg-comp/certs/`. Remember to set appropriate rights to the key file.
+{{{
+#!div style="font-size: 90%"
+{{{#!default
+cp /etc/grid-security/hostcert.pem /etc/qcg-comp/certs/qcgcert.pem
+cp /etc/grid-security/hostkey.pem /etc/qcg-comp/certs/qcgkey.pem
+chown qcg-comp /etc/qcg-comp/certs/qcgcert.pem
+chown qcg-comp /etc/qcg-comp/certs/qcgkey.pem 
+chmod 0600 /etc/qcg-comp/certs/qcgkey.pem
+}}}
+}}}
+=  DRMAA library =
+== Torque/PBS Professional ==
+Install DRMAA for Torque/PBS Pro using source package available at [http://apps.man.poznan.pl/trac/pbs-drmaa PBS DRMAA home page]
+
+
+== SLURM ==
+Install DRMAA for SLURM using source package available at [http://apps.man.poznan.pl/trac/slurm-drmaa SLURM DRMAA home page].
+{{{
+# install SLURM headers files
+apt-get install libslurm21-dev
+# get SLURM DRMAA
+wget http://apps.man.poznan.pl/trac/slurm-drmaa/downloads/slurm-drmaa-1.0.5.tar.gz
+tar -xzf slurm-drmaa-1.0.5.tar.gz
+cd slurm-drmaa-1.0.5
+# configure, make and install (by default DRMAA should be installed into /usr/local/
+./configure
+make 
+make install
+# test it!
+ /usr/local/bin/drmaa-run /bin/hostname
+
+}}}
+= Service configuration  =
+Edit the preinstalled service configuration file (`/etc/qcg-comp/qcg-comp`**d**`.xml`):
+{{{
+<?xml version="1.0" encoding="UTF-8"?>
+<sm:QCGCore
+        xmlns:sm="http://schemas.qoscosgrid.org/core/2011/04/config"
+        xmlns="http://schemas.qoscosgrid.org/comp/2011/04/config"
+        xmlns:smc="http://schemas.qoscosgrid.org/comp/2011/04/config"
+        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+        
+        <Configuration>
+                <sm:ModuleManager>
+                        <sm:Directory>/usr/lib/qcg-core/modules/</sm:Directory>
+                        <sm:Directory>/usr/lib/qcg-comp/modules/</sm:Directory>
+                </sm:ModuleManager>
+
+                <sm:Service xsi:type="qcg-compd" description="QCG Computing">
+                        <sm:Logger>
+                                <sm:Filename>/var/log/qcg-comp/qcg-compd.log</sm:Filename>
+                                <sm:Level>INFO</sm:Level>
+                        </sm:Logger>
+
+                        <sm:Transport>
+                                <sm:Module xsi:type="sm:ecm_gsoap.service">
+                                        <sm:Host>localhost</sm:Host>
+                                        <sm:Port>19000</sm:Port>
+                                </sm:Module>
+                                <sm:Module xsi:type="smc:qcg-comp-service"/>
+                        </sm:Transport>
+                        
+                        <sm:Authentication>
+        <sm:Module xsi:type="sm:atc_transport_gsi.service">
+                         <sm:X509CertFile>/etc/qcg-comp/certs/qcgcert.pem</sm:X509CertFile>
+                         <sm:X509KeyFile>/etc/qcg-comp/certs/qcgkey.pem</sm:X509KeyFile>
+        </sm:Module>
+                        </sm:Authentication>
+
+      <sm:Authorization>
+        <sm:Module xsi:type="sm:atz_mapfile">
+          <sm:Mapfile>/etc/grid-security/grid-mapfile</sm:Mapfile>
+        </sm:Module>
+      </sm:Authorization>
+
+
+                        <sm:Module xsi:type="submission_drmaa" path="/usr/local/lib/libdrmaa.so"/>
+
+      <!-- The jsdl filter module - uncomment module appropriate for your batch system -->
+      <!-- sm:Module xsi:type="pbs_jsdl_filter"/-->
+      <!-- sm:Module xsi:type="sge_jsdl_filter"/-->
+      <!-- sm:Module xsi:type="slurm_jsdl_filter"/-->
+      <!-- sm:Module xsi:type="lsf_jsdl_filter"/-->
+
+      <!-- The reservation module - uncomment module appropriate for your batch/scheduler system -->
+                        <!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_sge.py"/-->
+                        <!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_maui.py"/-->
+                        <!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_moab.py"/-->
+                        <!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_pbs.py"/-->
+                        <!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_slurm.py"/-->
+      <sm:Module xsi:type="atz_ardl_filter"/>
+
+      <sm:Module xsi:type="sm:general_python" path="/usr/lib/qcg-comp/modules/python/monitoring.py"/>
+                        
+                        <sm:Module xsi:type="notification_wsn">
+                                <sm:Module xsi:type="sm:ecm_gsoap.client" >
+                                                <sm:ServiceURL>http://localhost:19001/</sm:ServiceURL>
+                                                        <sm:Authentication>
+                                                                <sm:Module xsi:type="sm:atc_transport_http.client"/>
+                                                        </sm:Authentication>
+                                                <sm:Module xsi:type="sm:ntf_client"/>
+                                </sm:Module>
+                        </sm:Module>
+                                
+                        <sm:Module xsi:type="application_mapper">
+                                <ApplicationMapFile>/etc/qcg-comp/application_mapfile</ApplicationMapFile>
+                        </sm:Module>
+
+                        <Database>
+                                <DSN>qcg-comp</DSN>
+                                <User>qcg-comp</User>
+                                <Password>qcg-comp</Password>
+                        </Database>
+
+                        <UnprivilegedUser>qcg-comp</UnprivilegedUser>
+
+                        <FactoryAttributes>
+                                <CommonName>IT cluster</CommonName>
+                                <LongDescription>IT department cluster for public use</LongDescription>
+                        </FactoryAttributes>
+                </sm:Service>
+
+        </Configuration>
+</sm:QCGCore>
+}}}
+In most cases it should be enough to change only following elements:
+ `Transport/Module/Host` ::
+   the hostname of the machine where the service is deployed 
+ `Transport/Module/Authentication/Module/X509CertFile`  and  `Transport/Module/Authentication/Module/X509KeyFile` :: 
+  the service private key and X.509 certificate. Make sure that the key and certificate is owned by the `qcg-comp` user.  If you installed cert and key file in the recommended location you do not need to edit these fields.
+ `Module[type="smc:notification_wsn"]/PublishedBrokerURL` :: 
+  the external URL of the QCG-Notification service (You can do it later, i.e. after [http://apps.man.poznan.pl/trac/qcg-notification/wiki/InstallingUsingDeb installing the QCG-Notification service])
+ `Module[type="smc:notification_wsn"]/Module/ServiceURL` :: 
+  the localhost URL of the QCG-Notification service (You can do it later, i.e. after [http://apps.man.poznan.pl/trac/qcg-notification/wiki/InstallingUsingDeb installing the QCG-Notification service])
+ `Module[type="submission_drmaa"]/@path` ::
+  path to the DRMAA library (the `libdrmaa.so`). Also, if you installed the DRMAA library using provided SRC RPM you do not need to change this path.
+ `Database/Password` :: 
+  the `qcg-comp` database password  
+  `UseScratch` ::
+  set this to `true` if you set QCG_SCRATCH_DIR_ROOT in `sysconfig` so any job will be started from scratch directory (instead of the default home directory)
+ `FactoryAttributes/CommonName` :: 
+  a common name of the cluster (e.g. reef.man.poznan.pl). You can use any name that is unique among all systems (e.g. cluster name + domain name of your institution)
+ `FactoryAttributes/LongDescription` :: 
+  a human readable description of the cluster
+
+Moreover remember to uncomment  `jsdl_filter` and `reservation_python` modules (appropriate for your batch system).
+
+
+= Creating applications' script space =
+A common case for the QCG-Computing service is that an application is accessed using abstract app name rather than specifying absolute executable path. The application name/version to executbale path mappings are stored in the file `/etc/qcg-comp/application_mapfile`:
+
+{{{#!default
+cat /etc/qcg-comp/application_mapfile
+# ApplicationName ApplicationVersion Executable
+
+date * /bin/date
+LPSolve 5.5 /usr/local/bin/lp_solve
+}}}
+
+
+It is also common to provide here  wrapper scripts rather than target executables. The wrapper script can handle such aspects of the application lifetime like: environment initialization,  copying files from/to scratch storage and application monitoring. It is recommended to create separate directory for those wrapper scripts (e.g. the application partition) for an applications and add write permission to them to the QCG Developers group. This directory must be readable by all users and from every worker node (the application partition usually fullfils those requirements).
+
+{{{
+#!div style="font-size: 90%"
+{{{#!default
+mkdir /opt/exp_soft/qcg-app-scripts
+chown :qcg-dev /opt/exp_soft/qcg-app-scripts
+chmod g+rwx /opt/exp_soft/qcg-app-scripts
+}}}
+}}}
+
+More on [ApplicationScripts Application Scripts].
+= Note on the security model =
+The QCG-Computing can be configured with various authentication and authorization modules. However in the typical deployment we assume that the QCG-Computing is configured as in the above example, i.e.:
+* authentication is provided on basics of ''httpg'' protocol,
+* authorization is based on the local `grid-mapfile` mapfile.
+
+= Starting the service =
+As root type:
+{{{
+#!div style="font-size: 90%"
+{{{#!sh
+/etc/init.d/qcg-comp start
+}}}
+}}}
+
+The service logs can be found in:
+{{{#!sh
+/var/log/qcg-comp/qcg-compd.log
+}}}
+
+
+
+
+= Stopping the service =
+The service can be stopped using the following command:
+{{{
+#!div style="font-size: 90%"
+{{{#!sh
+/etc/init.d/qcg-comp stop
+}}}
+}}}
+
+= Verifying the installation =
+
+*  Edit the QCG-Computing client configuration file (`/etc/qcg-comp/qcg-comp.xml`):
+ *  set the `Host` and `Port` to reflects the changes in the service configuration file (`qcg-compd.xml`).
+{{{
+#!div style="font-size: 90%"
+{{{#!sh
+<?xml version="1.0" encoding="UTF-8"?>
+<sm:QCGCore
+       xmlns:sm="http://schemas.qoscosgrid.org/core/2011/04/config"
+       xmlns="http://schemas.qoscosgrid.org/comp/2011/04/config"
+       xmlns:smc="http://schemas.qoscosgrid.org/comp/2011/04/config"
+       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+  
+       <Configuration>
+               <sm:ModuleManager>
+                       <sm:Directory>/opt/qcg/lib/qcg-core/modules/</sm:Directory>
+                       <sm:Directory>/opt/qcg/lib/qcg-comp/modules/</sm:Directory>
+               </sm:ModuleManager>
+ 
+               <sm:Client xsi:type="qcg-comp" description="QCG-Computing client">
+                       <sm:Transport>
+                               <sm:Module xsi:type="sm:ecm_gsoap.client">
+                                       <sm:ServiceURL>httpg://frontend.example.com:19000/</sm:ServiceURL>
+                                       <sm:Authentication>
+                                               <sm:Module xsi:type="sm:atc_transport_gsi.client"/>
+                                       </sm:Authentication>
+                                       <sm:Module xsi:type="smc:qcg-comp-client"/>
+                               </sm:Module>
+                       </sm:Transport>
+               </sm:Client>
+       </Configuration>
+</sm:qcgCore>
+}}}
+}}}
+* Initialize your credentials:
+{{{
+#!div style="font-size: 90%"
+{{{#!sh
+grid-proxy-init -rfc
+Your identity: /O=Grid/OU=QosCosGrid/OU=PSNC/CN=Mariusz Mamonski
+Enter GRID pass phrase for this identity:
+Creating proxy .................................................................. Done
+Your proxy is valid until: Wed Apr  6 05:01:02 2012
+}}}
+}}}
+* Query the QCG-Computing service:
+{{{
+#!div style="font-size: 90%"
+{{{#!sh
+qcg-comp -G | xmllint --format - # the xmllint is used only to present the result in more pleasant way
+  
+<bes-factory:FactoryResourceAttributesDocument xmlns:bes-factory="http://schemas.ggf.org/bes/2006/08/bes-factory">
+    <bes-factory:IsAcceptingNewActivities>true</bes-factory:IsAcceptingNewActivities>
+    <bes-factory:CommonName>IT cluster</bes-factory:CommonName>
+    <bes-factory:LongDescription>IT department cluster for public   use</bes-factory:LongDescription>
+    <bes-factory:TotalNumberOfActivities>0</bes-factory:TotalNumberOfActivities>
+    <bes-factory:TotalNumberOfContainedResources>1</bes-factory:TotalNumberOfContainedResources>
+    <bes-factory:ContainedResource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="bes-factory:BasicResourceAttributesDocumentType">
+        <bes-factory:ResourceName>worker.example.com</bes-factory:ResourceName>
+        <bes-factory:CPUArchitecture>
+            <jsdl:CPUArchitectureName xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl">x86_32</jsdl:CPUArchitectureName>
+        </bes-factory:CPUArchitecture>
+        <bes-factory:CPUCount>4</bes-factory:CPUCount><bes-factory:PhysicalMemory>1073741824</bes-factory:PhysicalMemory>
+    </bes-factory:ContainedResource>
+    <bes-factory:NamingProfile>http://schemas.ggf.org/bes/2006/08/bes/naming/BasicWSAddressing</bes-factory:NamingProfile> 
+    <bes-factory:BESExtension>http://schemas.ogf.org/hpcp/2007/01/bp/BasicFilter</bes-  factory:BESExtension>
+    <bes-factory:BESExtension>http://schemas.qoscosgrid.org/comp/2011/04</bes-factory:BESExtension>
+    <bes-factory:LocalResourceManagerType>http://example.com/SunGridEngine</bes-factory:LocalResourceManagerType>
+    <smcf:NotificationProviderURL xmlns:smcf="http://schemas.qoscosgrid.org/comp/2011/04/factory">http://localhost:2211/</smcf:NotificationProviderURL>
+</bes-factory:FactoryResourceAttributesDocument>
+}}}
+}}}
+* Submit a sample job:
+{{{
+#!div style="font-size: 90%"
+{{{#!sh
+qcg-comp -c -J /usr/share/doc/qcg-comp-doc/examples/date.xml
+Activity Id: ccb6b04a-887b-4027-633f-412375559d73
+}}}
+}}}
+* Query it status:
+{{{
+#!div style="font-size: 90%"
+{{{#!sh
+qcg-comp -s -a ccb6b04a-887b-4027-633f-412375559d73
+status = Executing
+qcg-comp -s -a ccb6b04a-887b-4027-633f-412375559d73
+status = Executing
+qcg-comp -s -a ccb6b04a-887b-4027-633f-412375559d73
+status = Finished
+exit status = 0
+}}}
+}}}