| | 1 | = Introduction = |
| | 2 | The SMOA Computing (the successor of the OpenDSP project) is an open source service acting as a computing provider exposing on demand access to computing resources and jobs over the HPC Basic Profile compliant Web Services interface. In addition the SMOA Computing offers remote interface for Advance Reservations management. |
| | 3 | |
| | 4 | = Requirements = |
| | 5 | Before installing SMOA Computing make sure that following software is installed: |
| | 6 | |
| | 7 | == Dependend libraries == |
| | 8 | 1. libxml2 >= 2.6.27 |
| | 9 | 2. openssl >= 0.9.7 |
| | 10 | 3. unixODBC >= 2.2.0 |
| | 11 | 4. Python >= 2.4 |
| | 12 | 5. [http://activemq.apache.org/cms/ ActiveMQ-CPP] >= 3.0.1 (optional - needed only for [http://www.plgrid.pl/en PL-Grid] BAT accounting module) |
| | 13 | |
| | 14 | '''Note:''' If you are installing the libraries from binary packages check if those libraries were installed in '''devel''' versions (i.e. with headers files). |
| | 15 | |
| | 16 | '''Note:''' For Scientific Linux 5.3 (Boron) you can use the following [http://fury.man.poznan.pl/smoa-packages/sl/x86_64/ rpms] in order to satisfy libxml2 dependency. |
| | 17 | |
| | 18 | == Database Backend == |
| | 19 | For the database backend we recommends the [http://www.postgresql.org/ PostgreSQL], as the SMOA Computing is shipped with SQL script for the database scheme creation for the PostgreSQL. Check if ODBC drivers for the PostgreSQL were installed (i.e. if the <code>libodbcpsql.so</code> and <code>libodbcpsqlS.so</code> files are available in your system), if not install it independently. |
| | 20 | |
| | 21 | == DRMAA and ARES libraries == |
| | 22 | If you decided to use LSF as the LRMS system you need to install two external libraries: |
| | 23 | |
| | 24 | * DRMAA library for LSF (you can get it from [http://node2.qoscosgrid.man.poznan.pl/~qcg/download/lsf_drmaa-1.0.4.tar.gz here]) |
| | 25 | * ARES (Advance REServation) library for LSF (you can get it from [http://node2.qoscosgrid.man.poznan.pl/~qcg/download/lsf_ares-1.0.tar.gz here]) |
| | 26 | |
| | 27 | '''Note:''' Remember were the libraries where installed. You will need this information during [[SMOA_Computing#Service_configuration| Service configuration]]. |
| | 28 | |
| | 29 | If SGE: |
| | 30 | |
| | 31 | * DRMAA library is provided with standard SGE installation. |
| | 32 | * Instead of ARES library a Python module (installed by default with SMOA Computing) is used. Thus to make Advance Reservation capabilities accessible via SMOA Computing you will need Python interpreter (version 2.4 at least) together with the python library for XML processing (''lxml'' or ''elementtree''). |
| | 33 | |
| | 34 | if Torque with Maui scheduler: |
| | 35 | * DRMAA library can be obtained from the [http://sourceforge.net/projects/pbspro-drmaa/ PBS DRMAA sourceforge project]. After installation, as described in the library README file, you need either to: |
| | 36 | * configure Torque to keep information about completed jobs (e.g.: by setting: qmgr -c 'set server keep_completed = 60'). |
| | 37 | * configure the DRMAA library to use Torque logs. |
| | 38 | |
| | 39 | Sample configuration file of the DRMAA library (/etc/pbs_drmaa.conf): |
| | 40 | |
| | 41 | {{{ |
| | 42 | #!div style="font-size: 80%; background: #ffe" |
| | 43 | |
| | 44 | {{{#!default |
| | 45 | # pbs_drmaa.conf - Sample pbs_drmaa configuration file. |
| | 46 | wait_thread: 1, |
| | 47 | pbs_home: "/var/spool/pbs", |
| | 48 | pool_delay: 5, |
| | 49 | cache_job_state: 1, |
| | 50 | hello = lambda: "world" |
| | 51 | }}} |
| | 52 | |
| | 53 | }}} |
| | 54 | |
| | 55 | |
| | 56 | * Instead of ARES library a Python module (installed by default with SMOA Computing) is used. Thus to make Advance Reservation capabilities accessible via SMOA Computing you will need Python interpreter (version 2.4 at least) together with the python library for XML processing (''lxml'' or ''elementtree''). |
| | 57 | |
| | 58 | =Installation= |
| | 59 | == SMOA Core library == |
| | 60 | [http://fury.man.poznan.pl/smoa-downloads/smoa-core/smoa-core-latest.tar.gz SMOA Core] is an utility and interoperability layer for the all SMOA services (SMOA Computing and SMOA Notification in particular). As the GSI (Grid Security Infrastructure) is used to secure the transport layer you must compile the SMOA Core library with GSI support. After successful [[gridFTP]] installation (we recommend to use version 4.2.x of Globus Toolkit) all you need to do is to check if the <tt>GLOBUS_LOCATION</tt> environment variable is set correctly (and the <code>$GLOBUS_LOCATION/etc/globus-user-env.sh</code> file sourced). You must also choose globus flavor that you wish to compile with (e.g. <code>gcc64dbg</code> for a 64bit machine or <code>gcc32dbg</code> if you are installing the SMOA Core on a 32bit machine): |
| | 61 | |
| | 62 | $ . $GLOBUS_LOCATION/etc/globus-user-env.sh |
| | 63 | $ tar xzf smoa-core-latest.tar.gz |
| | 64 | $ cd smoa-core-2.0.4 |
| | 65 | $ ./configure --prefix=/opt/QCG/smoa --with-globus-flavor=gcc64dbg |
| | 66 | $ make |
| | 67 | $ sudo make install |
| | 68 | |
| | 69 | {{Note}} You may need to pass some additional options to <tt>./configure</tt> script, if some of the dependencies are installed in non-standard locations (type <tt>./configure --help</tt> for more info) |
| | 70 | |
| | 71 | == SMOA Computing service == |
| | 72 | |
| | 73 | |
| | 74 | After downloading the [http://fury.man.poznan.pl/smoa-downloads/smoa-comp/smoa-comp-latest.tar.gz source package] the SMOA Computing installation can be done in a few steps, e.g: |
| | 75 | |
| | 76 | $ tar xzf smoa-comp-latest.tar.gz |
| | 77 | $ cd smoa-comp-2.2 |
| | 78 | $ ./configure --prefix=/opt/QCG/smoa/ --with-smoa-core=/opt/QCG/smoa/ |
| | 79 | $ make |
| | 80 | $ sudo make install |
| | 81 | |
| | 82 | =Configuration= |
| | 83 | Before you start create a new system user (e.g. <code>smoa_comp</code>) that will be used by the service (and only the SMOA Computing service) to run in unprivileged mode: |
| | 84 | # useradd -d /opt/QCG/smoa/var/log/smoa-comp/ -M smoa_comp |
| | 85 | |
| | 86 | == Database setup == |
| | 87 | You need to create a database for SMOA Computing and appropriate tables using the provided <tt>smoa-comp-psql.sql</tt> script. It can be found in <tt>PREFIX/share/smoa-comp/db/</tt>. Currently only PostgreSQL is supported. |
| | 88 | |
| | 89 | * Create new database user (e.g. smoa_comp) authenticated via password (you will later provide this password in the SMOA Computing configuration file). This user must be allowed to create new databases. |
| | 90 | # su - postgres |
| | 91 | $ createuser -P smoa_comp |
| | 92 | $ exit |
| | 93 | |
| | 94 | * Create new database (e.g. smoa_comp) owned by the smoa_comp user: |
| | 95 | # su - smoa_comp |
| | 96 | $ createdb -U smoa_comp smoa_comp |
| | 97 | $ exit |
| | 98 | |
| | 99 | |
| | 100 | * Depending on the local PostgreSQL configuration you may need to edit the <code>pg_hba.conf</code> file (''host based authentication'' configuration file) and add the following lines as the two top rules: |
| | 101 | local smoa_comp smoa_comp md5 |
| | 102 | host smoa_comp smoa_comp 127.0.0.1/32 md5 |
| | 103 | : in order to enable password authentication for the user <code>smoa_comp</code> |
| | 104 | |
| | 105 | {{Note|}} You must reload the PostgreSQL server in order to make the changes visible e.g: |
| | 106 | |
| | 107 | # /etc/init.d/postgresql reload |
| | 108 | |
| | 109 | * Create database schema using provided script (<code>smoa-comp-psql.sq</code>) : |
| | 110 | |
| | 111 | $ psql -f /opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql -U smoa_comp |
| | 112 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:3: ERROR: table "states" does not exist |
| | 113 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:4: ERROR: table "jobs" does not exist |
| | 114 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:5: ERROR: table "jobs_acc" does not exist |
| | 115 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:6: ERROR: table "reservations" does not exist |
| | 116 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:7: ERROR: table "reservations_acc" does not exist |
| | 117 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:12: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "states_pkey" for table "states" |
| | 118 | CREATE TABLE |
| | 119 | INSERT 0 1 |
| | 120 | INSERT 0 1 |
| | 121 | INSERT 0 1 |
| | 122 | INSERT 0 1 |
| | 123 | INSERT 0 1 |
| | 124 | INSERT 0 1 |
| | 125 | INSERT 0 1 |
| | 126 | INSERT 0 1 |
| | 127 | INSERT 0 1 |
| | 128 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:53: NOTICE: CREATE TABLE will create implicit sequence "jobs_serial_seq" for serial column "jobs.serial" |
| | 129 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:53: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "jobs_pkey" for table "jobs" |
| | 130 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:53: NOTICE: CREATE TABLE / UNIQUE will create implicit index "jobs_drms_id_key" for table "jobs" |
| | 131 | CREATE TABLE |
| | 132 | CREATE INDEX |
| | 133 | CREATE INDEX |
| | 134 | CREATE INDEX |
| | 135 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:88: NOTICE: CREATE TABLE will create implicit sequence "jobs_acc_serial_seq" for serial column "jobs_acc.serial" |
| | 136 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:88: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "jobs_acc_pkey" for table "jobs_acc" |
| | 137 | CREATE TABLE |
| | 138 | CREATE INDEX |
| | 139 | CREATE INDEX |
| | 140 | CREATE INDEX |
| | 141 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:109: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "reservations_pkey" for table "reservations" |
| | 142 | CREATE TABLE |
| | 143 | CREATE INDEX |
| | 144 | psql:/opt/QCG/smoa/share/smoa-comp/db/smoa-comp-psql.sql:128: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "reservations_acc_pkey" for table "reservations_acc" |
| | 145 | CREATE TABLE |
| | 146 | CREATE INDEX |
| | 147 | |
| | 148 | * You need to create ODBC Data Source Name. You can do this by editing system-wide, i.e. /etc/odbc.ini, configuration file and adding new section. In this example we assume that the DSN (Database Source Name) will be smoa_comp, real database name is also smoa_comp and appropriate drivers for PostgreSQL database (i.e. the <code>libodbcpsql.so</code> and <code>libodbcpsqlS.so</code> files) are located in /usr/local/lib: |
| | 149 | |
| | 150 | [smoa_comp] |
| | 151 | Description = SMOA Computing database |
| | 152 | Driver = /usr/local/lib/libodbcpsql.so |
| | 153 | Setup = /usr/local/lib/libodbcpsqlS.so |
| | 154 | Database = smoa_comp |
| | 155 | Servername = localhost |
| | 156 | Port = 5432 |
| | 157 | ReadOnly = No |
| | 158 | |
| | 159 | {{Note|}} Verify if paths given in the ''Driver'' and ''Setup'' sections are valid. |
| | 160 | |
| | 161 | {{Note|}} The PostgreSQL odbc driver on many systems is distributed as a separate package (the <code>postgresql-odbc</code> or similar). Don't use the PostgreSQL driver shipped with the UnixODBC source tarball, as this one is known to be [https://bugzilla.redhat.com/show_bug.cgi?id=458708#c7 obsolete]. |
| | 162 | |
| | 163 | * It is recommended to use a command line client to access the database through ODBC to check if everything was configured correctly. For example, unixODBC provides an <code>isql</code> client: |
| | 164 | |
| | 165 | $ isql -v smoa_comp smoa_comp <password> |
| | 166 | +---------------------------------------+ |
| | 167 | | Connected! | |
| | 168 | | | |
| | 169 | | sql-statement | |
| | 170 | | help [tablename] | |
| | 171 | | quit | |
| | 172 | | | |
| | 173 | +---------------------------------------+ |
| | 174 | SQL>quit |
| | 175 | $ |
| | 176 | |
| | 177 | {{Note|}} It is recommended to run this command as user different than root (e.g. the smoa_comp user). |
| | 178 | |
| | 179 | == Service configuration == |
| | 180 | You can find the complete reference guide in the SMOA Computing manual (that can be found under the <code>doc/</code> directory). For convenience we provided below sample service configuration files (located in <code>PREFIX/etc/smoa-compd.xml</code> - do not mislead this file with the client configuration file which is named <code>smoa-comp.xml</code>) that you can use as templates: |
| | 181 | |
| | 182 | === Template configuration for the SMOA Computing service deployed on the top of LSF system === |
| | 183 | |
| | 184 | <?xml version="1.0" encoding="UTF-8"?> |
| | 185 | <sm:SMOACore |
| | 186 | xmlns:sm="http://schemas.smoa-project.com/core/2009/01/config" |
| | 187 | xmlns="http://schemas.smoa-project.com/comp/2009/01/config" |
| | 188 | xmlns:smc="http://schemas.smoa-project.com/comp/2009/01/config" |
| | 189 | xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> |
| | 190 | |
| | 191 | <Configuration> |
| | 192 | <sm:ModuleManager> |
| | 193 | <sm:Directory>/opt/QCG/smoa//lib/smoa-core/modules/</sm:Directory> |
| | 194 | <sm:Directory>/opt/QCG/smoa//lib/smoa-comp/modules/</sm:Directory> |
| | 195 | </sm:ModuleManager> |
| | 196 | |
| | 197 | <sm:Service xsi:type="smoa-compd" description="SMOA Computing"> |
| | 198 | <sm:Logger> |
| | 199 | <sm:Filename>/opt/QCG/smoa//var/log/smoa-comp/smoa-comp.log</sm:Filename> |
| | 200 | <sm:Level>INFO</sm:Level> |
| | 201 | </sm:Logger> |
| | 202 | |
| | 203 | <sm:Transport> |
| | 204 | <sm:Module xsi:type="sm:ecm_gsoap.service"> |
| | 205 | <sm:Host>frontend.example.com</sm:Host> |
| | 206 | <sm:Port>19000</sm:Port> |
| | 207 | <sm:KeepAlive>false</sm:KeepAlive> |
| | 208 | <sm:Authentication> |
| | 209 | <sm:Module xsi:type="sm:atc_transport_gsi.service"> |
| | 210 | <sm:X509CertFile>/opt/QCG/smoa/certs/smoacert.pem</sm:X509CertFile> |
| | 211 | <sm:X509KeyFile>/opt/QCG/smoa/certs/smoakey.pem</sm:X509KeyFile> |
| | 212 | </sm:Module> |
| | 213 | </sm:Authentication> |
| | 214 | <sm:Authorization> |
| | 215 | <sm:Module xsi:type="sm:atz_mapfile"> |
| | 216 | <sm:Mapfile>/etc/grid-security/grid-mapfile</sm:Mapfile> |
| | 217 | </sm:Module> |
| | 218 | </sm:Authorization> |
| | 219 | </sm:Module> |
| | 220 | <sm:Module xsi:type="smc:smoa-comp-service"/> |
| | 221 | </sm:Transport> |
| | 222 | |
| | 223 | <sm:Module xsi:type="submission_drmaa" path="/opt/QCG/LSF/lsf7.0.3/7.0/linux2.6-glibc2.3-x86/lib/libdrmaa.so"/> |
| | 224 | <sm:Module xsi:type="reservation_ares" path="/opt/QCG/smoa/lib/libares.so"/> |
| | 225 | |
| | 226 | <sm:Module xsi:type="lsf_jsdl_filter"/> |
| | 227 | <sm:Module xsi:type="smc:notification_wsn"> |
| | 228 | <sm:Module xsi:type="sm:ecm_gsoap.client" > |
| | 229 | <sm:ServiceURL>http://frontend.example.com:19005/</sm:ServiceURL> |
| | 230 | <sm:Authentication> |
| | 231 | <sm:Module xsi:type="sm:atc_transport_http.client"/> |
| | 232 | </sm:Authentication> |
| | 233 | <sm:Module xsi:type="sm:ntf_client"/> |
| | 234 | </sm:Module> |
| | 235 | </sm:Module> |
| | 236 | |
| | 237 | <sm:Module xsi:type="atz_ardl_filter"/> |
| | 238 | |
| | 239 | <sm:Module xsi:type="application_mapper"> |
| | 240 | <ApplicationMapFile>/opt/QCG/smoa/etc/application_mapfile</ApplicationMapFile> |
| | 241 | </sm:Module> |
| | 242 | |
| | 243 | <Database> |
| | 244 | <DSN>smoa_comp</DSN> |
| | 245 | <User>smoa_comp</User> |
| | 246 | <Password>smoa_comp</Password> |
| | 247 | </Database> |
| | 248 | |
| | 249 | <UnprivilegedUser>smoa_comp</UnprivilegedUser> |
| | 250 | <ReservationsAdministrator>lsfadmin</ReservationsAdministrator> |
| | 251 | <FactoryAttributes> |
| | 252 | <CommonName>IT cluster</CommonName> |
| | 253 | <LongDescription>IT department cluster for public use</LongDescription> |
| | 254 | <LocalResourceManagerType>http://example.com/LSF</LocalResourceManagerType> |
| | 255 | <ContainedResources xmlns:bes-factory="http://schemas.ggf.org/bes/2006/08/bes-factory" xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl"> |
| | 256 | <ContainedResource> |
| | 257 | <bes-factory:ResourceName>worker.example.com</bes-factory:ResourceName> |
| | 258 | <bes-factory:CPUArchitecture> |
| | 259 | <jsdl:CPUArchitectureName>x86_32</jsdl:CPUArchitectureName> |
| | 260 | </bes-factory:CPUArchitecture> |
| | 261 | <bes-factory:CPUCount>4</bes-factory:CPUCount> |
| | 262 | <bes-factory:PhysicalMemory>1073741824</bes-factory:PhysicalMemory> |
| | 263 | </ContainedResource> |
| | 264 | </ContainedResources> |
| | 265 | </FactoryAttributes> |
| | 266 | </sm:Service> |
| | 267 | </Configuration> |
| | 268 | </sm:SMOACore> |
| | 269 | |
| | 270 | === Template configuration for the SMOA Computing service deployed on the top of Sun Grid Engine === |
| | 271 | |
| | 272 | <?xml version="1.0" encoding="UTF-8"?> |
| | 273 | <sm:SMOACore |
| | 274 | xmlns:sm="http://schemas.smoa-project.com/core/2009/01/config" |
| | 275 | xmlns="http://schemas.smoa-project.com/comp/2009/01/config" |
| | 276 | xmlns:smc="http://schemas.smoa-project.com/comp/2009/01/config" |
| | 277 | xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> |
| | 278 | |
| | 279 | <Configuration> |
| | 280 | <sm:ModuleManager> |
| | 281 | <sm:Directory>/opt/QCG/smoa/lib/smoa-core/modules/</sm:Directory> |
| | 282 | <sm:Directory>/opt/QCG/smoa/lib/smoa-comp/modules/</sm:Directory> |
| | 283 | </sm:ModuleManager> |
| | 284 | |
| | 285 | <sm:Service xsi:type="smoa-compd" description="SMOA Computing"> |
| | 286 | <sm:Logger> |
| | 287 | <sm:Filename>/opt/QCG/smoa/var/log/smoa-comp/smoa-comp.log</sm:Filename> |
| | 288 | <sm:Level>INFO</sm:Level> |
| | 289 | </sm:Logger> |
| | 290 | |
| | 291 | <sm:Transport> |
| | 292 | <sm:Module xsi:type="sm:ecm_gsoap.service"> |
| | 293 | <sm:Host>frontend.example.com</sm:Host> |
| | 294 | <sm:Port>19000</sm:Port> |
| | 295 | <sm:KeepAlive>false</sm:KeepAlive> |
| | 296 | <sm:Authentication> |
| | 297 | <sm:Module xsi:type="sm:atc_transport_gsi.service"> |
| | 298 | <sm:X509CertFile>/opt/QCG/smoa/etc/certs/smoacert.pem</sm:X509CertFile> |
| | 299 | <sm:X509KeyFile>/opt/QCG/smoa/etc/certs/smoakey.pem</sm:X509KeyFile> |
| | 300 | </sm:Module> |
| | 301 | </sm:Authentication> |
| | 302 | <sm:Authorization> |
| | 303 | <sm:Module xsi:type="sm:atz_mapfile"> |
| | 304 | <sm:Mapfile>/etc/grid-security/grid-mapfile</sm:Mapfile> |
| | 305 | </sm:Module> |
| | 306 | </sm:Authorization> |
| | 307 | </sm:Module> |
| | 308 | <sm:Module xsi:type="smc:smoa-comp-service"/> |
| | 309 | </sm:Transport> |
| | 310 | |
| | 311 | <sm:Module xsi:type="submission_drmaa" path="/opt/sge/lib/lx24-x86/libdrmaa.so"/> |
| | 312 | <sm:Module xsi:type="reservation_python" path="/opt/QCG/smoa/lib/smoa-comp/modules/python/reservation_sge.py"/> |
| | 313 | |
| | 314 | <sm:Module xsi:type="sge_jsdl_filter"/> |
| | 315 | <sm:Module xsi:type="atz_ardl_filter"/> |
| | 316 | |
| | 317 | <sm:Module xsi:type="application_mapper"> |
| | 318 | <ApplicationMapFile>/opt/QCG/smoa/etc/application_mapfile</ApplicationMapFile> |
| | 319 | </sm:Module> |
| | 320 | |
| | 321 | <sm:Module xsi:type="smc:notification_wsn"> |
| | 322 | <sm:Module xsi:type="sm:ecm_gsoap.client" > |
| | 323 | <sm:ServiceURL>http://frontend.example.com:19005/</sm:ServiceURL> |
| | 324 | <sm:Authentication> |
| | 325 | <sm:Module xsi:type="sm:atc_transport_http.client"/> |
| | 326 | </sm:Authentication> |
| | 327 | <sm:Module xsi:type="sm:ntf_client"/> |
| | 328 | </sm:Module> |
| | 329 | </sm:Module> |
| | 330 | |
| | 331 | <Database> |
| | 332 | <DSN>smoa_comp</DSN> |
| | 333 | <User>smoa_comp</User> |
| | 334 | <Password>smoa_comp</Password> |
| | 335 | </Database> |
| | 336 | |
| | 337 | <UnprivilegedUser>smoa_comp</UnprivilegedUser> |
| | 338 | <OnlyOwnerCanGetStatus>false</OnlyOwnerCanGetStatus> |
| | 339 | <OnlyOwnerCanGetDocument>false</OnlyOwnerCanGetDocument> |
| | 340 | <ReservationsAdministrator>sgeadmin</ReservationsAdministrator> |
| | 341 | |
| | 342 | <FactoryAttributes> |
| | 343 | <CommonName>IT cluster</CommonName> |
| | 344 | <LongDescription>IT department cluster for public use</LongDescription> |
| | 345 | <LocalResourceManagerType>http://example.com/SunGridEngine</LocalResourceManagerType> |
| | 346 | <ContainedResources xmlns:bes-factory="http://schemas.ggf.org/bes/2006/08/bes-factory" xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl"> |
| | 347 | <ContainedResource> |
| | 348 | <bes-factory:ResourceName>worker.example.com</bes-factory:ResourceName> |
| | 349 | <bes-factory:CPUArchitecture> |
| | 350 | <jsdl:CPUArchitectureName>x86_32</jsdl:CPUArchitectureName> |
| | 351 | </bes-factory:CPUArchitecture> |
| | 352 | <bes-factory:CPUCount>4</bes-factory:CPUCount> |
| | 353 | <bes-factory:PhysicalMemory>1073741824</bes-factory:PhysicalMemory> |
| | 354 | </ContainedResource> |
| | 355 | </ContainedResources> |
| | 356 | </FactoryAttributes> |
| | 357 | </sm:Service> |
| | 358 | </Configuration> |
| | 359 | </sm:SMOACore> |
| | 360 | |
| | 361 | === Template configuration for the SMOA Computing service deployed on the top of Torque (using Maui scheduler) === |
| | 362 | <?xml version="1.0" encoding="UTF-8"?> |
| | 363 | <sm:SMOACore |
| | 364 | xmlns:sm="http://schemas.smoa-project.com/core/2009/01/config" |
| | 365 | xmlns="http://schemas.smoa-project.com/comp/2009/01/config" |
| | 366 | xmlns:smc="http://schemas.smoa-project.com/comp/2009/01/config" |
| | 367 | xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> |
| | 368 | |
| | 369 | <Configuration> |
| | 370 | <sm:ModuleManager> |
| | 371 | <sm:Directory>/opt/QCG/smoa/lib/smoa-core/modules/</sm:Directory> |
| | 372 | <sm:Directory>/opt/QCG/smoa/lib/smoa-comp/modules/</sm:Directory> |
| | 373 | </sm:ModuleManager> |
| | 374 | |
| | 375 | <sm:Service xsi:type="smoa-compd" description="SMOA Computing"> |
| | 376 | <sm:Logger> |
| | 377 | <sm:Filename>/opt/QCG/smoa/var/log/smoa-comp/smoa-comp.log</sm:Filename> |
| | 378 | <sm:Level>INFO</sm:Level> |
| | 379 | </sm:Logger> |
| | 380 | |
| | 381 | <sm:Transport> |
| | 382 | <sm:Module xsi:type="sm:ecm_gsoap.service"> |
| | 383 | <sm:Host>frontend.example.com</sm:Host> |
| | 384 | <sm:Port>19000</sm:Port> |
| | 385 | <sm:KeepAlive>false</sm:KeepAlive> |
| | 386 | <sm:Authentication> |
| | 387 | <sm:Module xsi:type="sm:atc_transport_gsi.service"> |
| | 388 | <sm:X509CertFile>/opt/QCG/smoa/etc/certs/smoacert.pem</sm:X509CertFile> |
| | 389 | <sm:X509KeyFile>/opt/QCG/smoa/etc/certs/smoakey.pem</sm:X509KeyFile> |
| | 390 | </sm:Module> |
| | 391 | </sm:Authentication> |
| | 392 | <sm:Authorization> |
| | 393 | <sm:Module xsi:type="sm:atz_mapfile"> |
| | 394 | <sm:Mapfile>/etc/grid-security/grid-mapfile</sm:Mapfile> |
| | 395 | </sm:Module> |
| | 396 | </sm:Authorization> |
| | 397 | </sm:Module> |
| | 398 | <sm:Module xsi:type="smc:smoa-comp-service"/> |
| | 399 | </sm:Transport> |
| | 400 | |
| | 401 | <sm:Module xsi:type="submission_drmaa" path="/opt/QCG/smoa/lib/libdrmaa.so"/> |
| | 402 | <sm:Module xsi:type="reservation_python" path="/opt/QCG/smoa/lib/smoa-comp/modules/python/reservation_maui.py"/> |
| | 403 | |
| | 404 | <sm:Module xsi:type="pbs_jsdl_filter"/> |
| | 405 | <sm:Module xsi:type="atz_ardl_filter"/> |
| | 406 | |
| | 407 | <sm:Module xsi:type="application_mapper"> |
| | 408 | <ApplicationMapFile>/opt/QCG/smoa/etc/application_mapfile</ApplicationMapFile> |
| | 409 | </sm:Module> |
| | 410 | |
| | 411 | <sm:Module xsi:type="smc:notification_wsn"> |
| | 412 | <sm:Module xsi:type="sm:ecm_gsoap.client" > |
| | 413 | <sm:ServiceURL>http://frontend.example.com:19005/</sm:ServiceURL> |
| | 414 | <sm:Authentication> |
| | 415 | <sm:Module xsi:type="sm:atc_transport_http.client"/> |
| | 416 | </sm:Authentication> |
| | 417 | <sm:Module xsi:type="sm:ntf_client"/> |
| | 418 | </sm:Module> |
| | 419 | </sm:Module> |
| | 420 | |
| | 421 | <sm:Module xsi:type="sm:general_python" path="/opt/QCG/smoa/lib/smoa-comp/modules/python/monitoring.py"/> |
| | 422 | |
| | 423 | <Database> |
| | 424 | <DSN>smoa_comp</DSN> |
| | 425 | <User>smoa_comp</User> |
| | 426 | <Password>smoa_comp</Password> |
| | 427 | </Database> |
| | 428 | |
| | 429 | <UnprivilegedUser>smoa_comp</UnprivilegedUser> |
| | 430 | <ReservationsAdministrator>maui</ReservationsAdministrator> |
| | 431 | <FactoryAttributes> |
| | 432 | <CommonName>frontend.example.com</CommonName> |
| | 433 | <LongDescription>IT department cluster for public use</LongDescription> |
| | 434 | </FactoryAttributes> |
| | 435 | </sm:Service> |
| | 436 | </Configuration> |
| | 437 | </sm:SMOACore> |
| | 438 | In most cases it should be enough to change only following elements: |
| | 439 | ; ''Database'' : the contact data for the Database created in the previously steps |
| | 440 | ; ''Transport/Module/Host'' : the hostname of the machine where the service is deployed |
| | 441 | ; ''Transport/Module/Authentication/Module/X509CertFile'' and ''Transport/Module/Authentication/Module/X509KeyFile'' : - the service private key and X.509 certificate (consult the [http://www.globus.org/toolkit/docs/4.0/security/prewsaa/rn01re02.html Globus User Gide] on how to generate service certificate request or use the host certificate/key pair). Make sure that the key and certificate is owned by the <code>smoa_comp</code> user and the private key is not password protected (generating certificate with the <code>-service</code> option implies this). |
| | 442 | ; ''Module[type="smc:notification_wsn"]/Module/ServiceURL'' : the URL of the [[SMOA_Notification|SMOA Notification Service]] (You can do it later, i.e. after installing the SMOA Notification service) |
| | 443 | ; ''FactoryAttributes/ContainedResources'' : list of the all worker nodes managed by the LRMS. You should provide at least hostnames (''<bes-factory:ResourceName>'') and cpus count (''<bes-factory:CPUCount>'') for every worker node in your cluster. If you are using Torque batch system this information can be provided automatically by enabling ''python/monitoring.py'' module. |
| | 444 | ; Module[type="submission_drmaa"]/@path : path to the DRMAA library (the <code>libdrmaa.so</code> - search for this file within the LRMS lib directory. e.g. <code>/opt/sge/lib/</code> or <code>/opt/QCG/LSF/lsf7.0.3/7.0/linux2.6-glibc2.3-x86_64/lib/</code>) |
| | 445 | ; Module[type="reservation_ares"]/@path : path to the ARES library (LSF only) |
| | 446 | ; Module[type="reservation_python"]/@path : path to the Python reservation module (SGE only) - only if you installed SMOA Computing in location other than /opt/QCG/smoa. |
| | 447 | ; ReservationsAdministrator : the user authorized to create and manage reservations. Usually the LRMS administrator user (like <code>sgeadmin</code> in SGE or <code>lsfadmin</code> in LSF, or any user other than root given in ADMIN* sections of the Maui config file) has sufficient rights. |
| | 448 | |
| | 449 | {{Note}} Later in this manual you will read that the [[GRMS]] credentials must be mapped to this user (see [[GRMS#Resource_Providers_setup%23Resource%20Providers%20setup| GRMS Resource Providers setup]]). |
| | 450 | {{Security Note}} |
| | 451 | |
| | 452 | Finally make this configuration file readable '''only''' by the <code>smoa_comp</code> user (in order to protect database password): |
| | 453 | chmod 0600 /opt/QCG/smoa/etc/smoa-compd.xml |
| | 454 | |
| | 455 | == Configuring BAT accounting module (PL-Grid only) == |
| | 456 | In order to report resource usage to the central PL-Grid accounting service you must enable the <code>bat_updater</code> module. You can do this by including the following snippet in the configuration (just before the <code>Database</code> section): |
| | 457 | <sm:Module xsi:type="bat_updater"> |
| | 458 | <BATServiceURL>tcp://acct.grid.cyf-kr.edu.pl:61616</BATServiceURL> |
| | 459 | <SiteName>psnc-smoa-plgrid</SiteName> |
| | 460 | <QueueName>test-jobs</QueueName> |
| | 461 | </sm:Module> |
| | 462 | |
| | 463 | where: |
| | 464 | ;BATServiceURL : URL of the BAT accounting service |
| | 465 | ;SiteName : local site name as reported to the BAT service |
| | 466 | ;QueueName : queue name to which report usage data |
| | 467 | |
| | 468 | = Note on the security model = |
| | 469 | The SMOA Computing can be configured with various authentication and authorization modules. However in the typical deployment we assume that the SMOA Computing is configured as in the above example, i.e.: |
| | 470 | * authentication is provided on basics of ''httpg'' protocol |
| | 471 | * authorization is based on the local <code>grid-mapfile</code> mapfile (see [[GridFTP#Users_configuration| Users configuration]]). |
| | 472 | |
| | 473 | =Starting the service= |
| | 474 | As root type: |
| | 475 | |
| | 476 | # /opt/QCG/smoa/sbin/smoa-compd |
| | 477 | |
| | 478 | or if you want the service not to daemonize and print all logs to console type: |
| | 479 | |
| | 480 | # /opt/QCG/smoa/sbin/smoa-compd -d |
| | 481 | |
| | 482 | otherwise the logs can be found in <code>/opt/QCG/smoa/var/log/smoa-comp/smoa-comp.log</code>. |
| | 483 | |
| | 484 | {{Note}} Before starting the <code>smoa-compd</code> service make sure that all environment variables needed to contact the LRMS system are set (i.e. the <code>SGE_ROOT</code> for SGE or the whole <code>profile.lsf</code> file sourced in case of LSF) |
| | 485 | |
| | 486 | =Stopping the service= |
| | 487 | The service is stopped by sending SIGTERM signal, e.g.: |
| | 488 | # pkill smoa-compd |
| | 489 | |
| | 490 | =Verifying the installation= |
| | 491 | |
| | 492 | * For convenience you can add the <code>/opt/QCG/smoa/bin</code> to your <code>PATH</code> variable. |
| | 493 | * Edit the SMOA Computing client configuration file (<code>PREFIX/etc/smoa-comp.xml</code>): |
| | 494 | ** set the ''Host'' and ''Port'' to reflects the changes in the service configuration file (<code>smoa-compd.xml</code>), |
| | 495 | ** change the authentication module type from <code>sm:atc_transport_http.client</code> to <code>sm:atc_transport_gsi.client</code>, |
| | 496 | |
| | 497 | <?xml version="1.0" encoding="UTF-8"?> |
| | 498 | <sm:SMOACore |
| | 499 | xmlns:sm="http://schemas.smoa-project.com/core/2009/01/config" |
| | 500 | xmlns="http://schemas.smoa-project.com/comp/2009/01/config" |
| | 501 | xmlns:smc="http://schemas.smoa-project.com/comp/2009/01/config" |
| | 502 | xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> |
| | 503 | |
| | 504 | <Configuration> |
| | 505 | <sm:ModuleManager> |
| | 506 | <sm:Directory>/opt/QCG/smoa/lib/smoa-core/modules/</sm:Directory> |
| | 507 | <sm:Directory>/opt/QCG/smoa//lib/smoa-comp/modules/</sm:Directory> |
| | 508 | </sm:ModuleManager> |
| | 509 | |
| | 510 | <sm:Client xsi:type="smoa-comp" description="SMOA Computing client"> |
| | 511 | <sm:Transport> |
| | 512 | <sm:Module xsi:type="sm:ecm_gsoap.client"> |
| | 513 | <sm:ServiceURL>httpg://frontend.example.com:19000/</sm:ServiceURL> |
| | 514 | <sm:Authentication> |
| | 515 | <sm:Module xsi:type="sm:atc_transport_gsi.client"/> |
| | 516 | </sm:Authentication> |
| | 517 | <sm:Module xsi:type="smc:smoa-comp-client"/> |
| | 518 | </sm:Module> |
| | 519 | </sm:Transport> |
| | 520 | </sm:Client> |
| | 521 | </Configuration> |
| | 522 | </sm:SMOACore> |
| | 523 | |
| | 524 | * Initialize your credentials: |
| | 525 | |
| | 526 | $ grid-proxy-init |
| | 527 | Your identity: /O=Grid/OU=QosCosGrid/OU=PSNC/CN=Mariusz Mamonski |
| | 528 | Enter GRID pass phrase for this identity: |
| | 529 | Creating proxy .................................................................. Done |
| | 530 | Your proxy is valid until: Wed Sep 16 05:01:02 2009 |
| | 531 | |
| | 532 | * Query the SMOA Computing service: |
| | 533 | $ smoa-comp -G | xmllint --format - # the xmllint is used only to present the result in more pleasant way |
| | 534 | |
| | 535 | <bes-factory:FactoryResourceAttributesDocument xmlns:bes-factory="http://schemas.ggf.org/bes/2006/08/bes-factory"> |
| | 536 | <bes-factory:IsAcceptingNewActivities>true</bes-factory:IsAcceptingNewActivities> |
| | 537 | <bes-factory:CommonName>IT cluster</bes-factory:CommonName> |
| | 538 | <bes-factory:LongDescription>IT department cluster for public use</bes-factory:LongDescription> |
| | 539 | <bes-factory:TotalNumberOfActivities>0</bes-factory:TotalNumberOfActivities> |
| | 540 | <bes-factory:TotalNumberOfContainedResources>1</bes-factory:TotalNumberOfContainedResources> |
| | 541 | <bes-factory:ContainedResource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="bes-factory:BasicResourceAttributesDocumentType"> |
| | 542 | <bes-factory:ResourceName>worker.example.com</bes-factory:ResourceName> |
| | 543 | <bes-factory:CPUArchitecture> |
| | 544 | <jsdl:CPUArchitectureName xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl">x86_32</jsdl:CPUArchitectureName> |
| | 545 | </bes-factory:CPUArchitecture> |
| | 546 | <bes-factory:CPUCount>4</bes-factory:CPUCount><bes-factory:PhysicalMemory>1073741824</bes-factory:PhysicalMemory> |
| | 547 | </bes-factory:ContainedResource> |
| | 548 | <bes-factory:NamingProfile>http://schemas.ggf.org/bes/2006/08/bes/naming/BasicWSAddressing</bes-factory:NamingProfile> |
| | 549 | <bes-factory:BESExtension>http://schemas.ogf.org/hpcp/2007/01/bp/BasicFilter</bes- factory:BESExtension> |
| | 550 | <bes-factory:BESExtension>http://schemas.smoa-project.com/comp/2009/01</bes-factory:BESExtension> |
| | 551 | <bes-factory:LocalResourceManagerType>http://example.com/SunGridEngine</bes-factory:LocalResourceManagerType> |
| | 552 | <smcf:NotificationProviderURL xmlns:smcf="http://schemas.smoa-project.com/comp/2009/01/factory">http://localhost:2211/</smcf:NotificationProviderURL> |
| | 553 | </bes-factory:FactoryResourceAttributesDocument> |
| | 554 | |
| | 555 | * Submit a sample job: |
| | 556 | $ smoa-comp -c -J /opt/QCG/smoa/share/smoa-comp/doc/examples/jsdl/sleep.xml |
| | 557 | Activity Id: ccb6b04a-887b-4027-633f-412375559d73 |
| | 558 | |
| | 559 | * Query it status: |
| | 560 | $ smoa-comp -s -a ccb6b04a-887b-4027-633f-412375559d73 |
| | 561 | status = Executing |
| | 562 | $ smoa-comp -s -a ccb6b04a-887b-4027-633f-412375559d73 |
| | 563 | status = Executing |
| | 564 | $ smoa-comp -s -a ccb6b04a-887b-4027-633f-412375559d73 |
| | 565 | status = Finished |
| | 566 | exit status = 0 |
| | 567 | |
| | 568 | * Create an advance reservation: |
| | 569 | :* copy the provided sample reservation description file (expressed in ARDL - Advance Reservation Description Language) |
| | 570 | $ cp /opt/QCG/smoa/share/smoa-comp/doc/examples/ardl/oneslot.xml oneslot.xml |
| | 571 | |
| | 572 | :* Edit the <code>oneslot.xml</code> and modify the ''StartTime'' and ''EndTime'' to dates that are in the near future, |
| | 573 | :* Create a new reservation: |
| | 574 | $ smoa-comp -c -D oneslot.xml |
| | 575 | Reservation Id: aab6b04a-887b-4027-633f-412375559d7d |
| | 576 | :* List all reservations: |
| | 577 | $ smoa-comp -l |
| | 578 | Reservation Id: aab6b04a-887b-4027-633f-412375559d7d |
| | 579 | Total number of reservations: 1 |
| | 580 | :* Check which hosts where reserved: |
| | 581 | $ smoa-comp -s -r aab6b04a-887b-4027-633f-412375559d7d |
| | 582 | Reserved hosts: |
| | 583 | worker.example.com[used=0,reserved=1,total=4] |
| | 584 | :* In the end delete the reservation: |
| | 585 | $ smoa-comp -t -r aab6b04a-887b-4027-633f-412375559d7d |
| | 586 | Reservation terminated. |
