Changes between Version 17 and Version 18 of client_user_guide

Show
Ignore:
Timestamp:
08/07/13 16:52:15 (11 years ago)
Author:
piontek
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • client_user_guide

    v17 v18  
    4646* QCG_ENV_SKIP_KEY_CHECK - If the variable is not empty client do not check the correctness of the key format. Must be check if the client is configured to use the proxy and the key/certificate are not provided. 
    4747 
    48 = Job Profile = 
    49 Every experiment submitted to QCG-Broker has to be described by XML-based document called ''Job Profile''. The structure of ''Job Profile'' is formalized by [[http://node2.qoscosgrid.man.poznan.pl/~piontek/qcg-broker/QCGJobDescriptionSchema.xsd|Job Profile schema]]. 
    50  
    51 Examples of Job Profiles describing basic use cases are distributed with QCG-Broker and can be found in `<CLIENT_DIR>/examples directory`. 
    52  
    53 = Usage = 
    54  
    55 The QCG-Groker command-line java based client can operate in two modes: 
    56  
    57 * '''batch mode''' – that executes single operation with arguments passed directly to the client during its invocation. The batch mode allows to use the client in any kind of scripts mostly in cases when the processing of output is needed to steer the experiment, 
    58 * '''console mode''' – that works similar to shell console in which user can type in lines with operations and arguments to be executed by service. The console mode gives additional useful features like aliases, history accessible by arrows-keys, creation and management of user proxy, help functionality.  
    59  
    60 The usage of the client depends on the mode: 
    61 * for batch mode: "`qcg-client OPRATION [ARG1 .. ARGn]`" 
    62 * for console mode: "`qcg-client -console`" and then user is prompted to type in lines in format "`OPERATION [ARG1 .. ARGn]`" to be processed by client.  
    63  
    64 '''IMPORTANT:''' To secure the communication between client and service and to delegate user's privileges to the service client needs access to user's proxy certificate. 
    65  
    66 == Operations == 
    67  
    68 Regardless from the mode the QCG-Broker java based command-line client supports following list of operations: 
    69  
    70 ||= '''Operation''' =||= '''Arguments''' =||= '''Description''' =|| 
    71 ||='''submit_job''' =|| `<desc_file>` `[QCG_SIMPLE, QCG or JSDL]` || submits a job to be executed. The description of job can be expressed either in native QCG-Broker language (the default one) or in JSDL one. '''In case when the job to be submitted is described in JSDL format the type (JSDL) must be explicitly specified'''. If the description is valid client returns to the user a globally unique job identifier, which unambiguously identifies the job in the system.  QCG defines jobs as a sets of dependent tasks that constitute a logical whole (workflow). Each task is executed by system only if all tasks it depends on are in specified by the user states. ||  
    72 ||='''list_jobs''' =|| `[<limit>]` `[<status>]` || lists jobs belonging to the `user`. It is possible either to `limit` number of jobs or to display only ones in given state. All possible states are listed below  the table. || 
    73 ||='''list_user_jobs''' =|| `<user>` `[<limit>]` `[<status>]` || lists jobs belonging to the given `user`. The functionality is destined for administrative purposes. || 
    74 ||='''test_description''' =|| `<desc_file>` `[QCG or JSDL]` || validates job description ||  
    75 ||='''translate_description''' =|| `<desc_file>` JSDL || translates job description to native QCG-Broker one || 
    76 ||='''job_info''' =|| `<jobId>` `[<showJobDesc>]` || return complex information about the given job. If the `showJobDesc is` `true` the job description is shown. || 
    77 ||='''cancel_job''' =|| `<jobId>` || cancels execution of the given job || 
    78 ||='''commit_job''' =|| `<jobId>` || allows to approve the job submitted with two phase commit mechanism to be processed by the system. The two phase commit mechanism can be used to register notifications before the processing of the job will be started by broker. || 
    79 ||='''list_tasks''' =|| `<jobId>` `[<status>]` || lists tasks belonging to given job. Optionally it is possible to specify the task's status. Possible task statuses are listed below the table. || 
    80 ||='''tasks_statuses''' =|| `<jobId>` `[<summary>]` || lists tasks constituting the given job with their statuses. If the `summary` argument is `true` some additionall statistics is displayed. || 
    81 ||='''register_job_notification''' || `<jobId>` `<url>` || registers notification consumer for the given job || 
    82 ||='''list_job_notifications''' =|| `<jobId>` || lists notifications registered for the given job || 
    83 ||='''register_tasks_notification''' =|| `<jobId>` `<url>` || register notification for all tasks of the given job || 
    84 ||='''monitor_job''' =|| `<jobId>` `[<interval>]` || monitors status changes of tasks belonging to given job. The `interval` argument determines delay in seconds between next status checks. || 
    85 ||='''monitor_task''' =|| `<jobId>` `<taskId>` `[<interval>]` || monitors status changes of allocations belonging to the given tasks. The `interval` argument determines delay in seconds between next status checks. || 
    86 ||='''task_info''' =|| `<jobId>` `<taskId>` `[<showDesc>` `[<limit>]]` || displays information about the given task. If the `showDesc` is `true` the task description is shown. If the `limit` argument is specified the history of the task is limited to given value. =|| 
    87 ||='''register_task_notification''' =|| `<jobId>` `<taskId>` `<url>` || registers task's notification consumer || 
    88 ||='''list_task_notifications''' =|| `<jobId>` `<taskId>` || lists task's notifications || 
    89 ||='''cancel_task''' =|| `<jobId>` `<taskId>` || cancels execution of the given task || 
    90 ||='''commit_task''' =|| `<jobId>` `<taskId>` || commits the given task to be processed by the system || 
    91 ||='''reserve_resources''' =|| `[<taskId>]` `<job_desc>` (QCG or JSDL) || reserve resources that meet either the wole job or given task requirements. The reservation identifier is returned. This functionality is not implemented yet! || 
    92 ||='''reservation_info''' =|| `<reservationId>` || return complex information concerning the given reservation: list of reserved resources, local identifiers of reservations, reservation time slot. This functionality id not implemented yet! || 
    93 ||='''cancel_reservation''' =|| `<reservationId>` || releases reserved resources. This functionality is not implemented yet! || 
    94  
    95  
    96 List of Job statuses: 
    97     * UNCOMMITTED - the job was submitted with two phase commit option and waits to be committed, 
    98     * SUBMITTED – the job was submitted to the system and is executed by the system, 
    99     * SUSPENDED – the job was suspended, 
    100     * ACTIVE – the job is active, at least one task is processed, 
    101     * FINISHED – the job was completed, 
    102     * FAILED – the job (at least one crucial task belonging to the job) failed 
    103     * CANCELED – the job was canceled by the user, 
    104     * BROKEN - one or more of crucial tasks failed, system waits until active tasks will finish and change the status of the job to FAILED.|  
    105  
    106 List of Task statuses: 
    107     * UNSUBMITTED – the task cannot be started because of dependencies, 
    108     * UNCOMMITED - the task waits to be committed, 
    109     * QUEUED – the task was put into the queue and waits for execution, 
    110     * PREPROCESSING – system makes some actions needed to start the task (looks for the resource, stages in files), 
    111     * PENDING – the task is pending in the queueing-system, 
    112     * RUNNING – the task is active, 
    113     * STOPPED – the task was finished or was checkpointed, but system did not start staging out files, 
    114     * POSTPROCESSING – system makes some actions needed to complete the task, for example stages out files, cleares working environment, etc., 
    115     * FINISHED – the task was completed, 
    116     * SUSPENDED – the task was suspended, 
    117     * FAILED – the task failed, 
    118     * CANCELED – the task was canceled by the user. 
    119 == Usage examples == 
    120  
    121 The example presented below is the QCG-Broker Job Profile describing the parameter-sweep experiment executing a set of UNIX calendar tasks for the predefined space of "month" parameter. 
    122  
    123 {{{ 
    124 #!div style="font-size: 90%" 
    125 {{{#!xml 
    126 <qcgJob appId="calendar_example"> 
    127         <task persistent="true" taskId="calendar"> 
    128                 <execution type="single"> 
    129                         <executable> 
    130                                 <execFile> 
    131                                         <file> 
    132                                                 <location type="URL">file:////usr/bin/cal</location> 
    133                                         </file> 
    134                                 </execFile> 
    135                         </executable> 
    136                         <arguments> 
    137                                 <value>${PS_month}</value> 
    138                                 <value>2010</value> 
    139                         </arguments> 
    140                         <stdout> 
    141                                 <file> 
    142                                         <location type="URL">${TASK_DIR}/stdout.txt</location> 
    143                                 </file> 
    144                         </stdout> 
    145                 </execution> 
    146                 <parametersSweep> 
    147                         <parameter> 
    148                                 <name>month</name> 
    149                                 <value> 
    150                                         <loop> 
    151                                                 <start>1</start> 
    152                                                 <end>12</end> 
    153                                                 <step>1</step> 
    154                                                 <except> 
    155                                                         <value>3</value> 
    156                                                         <value>6</value> 
    157                                                 </except> 
    158                                         </loop> 
    159                                 </value> 
    160                         </parameter> 
    161                 </parametersSweep> 
    162         </task> 
    163 </qcgJob> 
    164 }}} 
    165 }}} 
    166  
    167 === submit_job === 
    168 * `submit_job <job_profile>` - submits job. `<job_profile>` must be the path to the file containing the Job Profile. 
    169 {{{ 
    170 #!div style="font-size: 90%" 
    171 {{{#!sh 
    172 qcg-client submit_job ../examples/usecase8.xml QCG 
    173 Your identity: C=PL,O=GRID,O=PSNC,CN=Tomasz Piontek 
    174 Creating proxy, please wait... 
    175 Proxy verify OK 
    176 Your proxy is valid until Tue May 17 02:55:47 CEST 2011 
    177 UserDN = /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek 
    178 ProxyLifetime = 0 Days 11 Hours 59 Minutes 57 Seconds 
    179  
    180 jobId = 1305550554579_calendar_example_5366 
    181 }}} 
    182 }}} 
    183  
    184 === list_jobs === 
    185 * `list_jobs [status] [limit]` - lists jobs. Optionally it is possible to specify status or limits the output to some number of recent jobs. 
    186 {{{ 
    187 #!div style="font-size: 90%" 
    188 {{{#!sh 
    189 qcg-client list_jobs 5 
    190 UserDN = /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek 
    191 ProxyLifetime = 0 Days 11 Hours 58 Minutes 47 Seconds 
    192  
    193 Number of jobs: 5 
    194 List of jobs:  
    195 1301904727887_calendar_example_1403 
    196 1304020897352_calendar_example_4252 
    197 1304065465905_calendar_example_9526 
    198 1305287977790_calendar_example_4779 
    199 1305550554579_calendar_example_5366 
    200 }}} 
    201 }}} 
    202  
    203 === job_info === 
    204 * `job_info <jobId> <showJobProfile>` - displays information about the given job. The `<showJobProfile>` argument of boolean type specifies whether the Job Profile should be displayed or not. 
    205 {{{ 
    206 #!div style="font-size: 90%" 
    207 {{{#!sh 
    208 qcg-client job_info 1305550554579_calendar_example_5366 false 
    209 UserDN = /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek 
    210 ProxyLifetime = 0 Days 11 Hours 58 Minutes 18 Seconds 
    211  
    212 UserDN: /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek 
    213 Project:  
    214 Status: FINISHED 
    215 StatusDesc:  
    216 SubmissionTime: Mon May 16 14:55:54 CEST 2011 
    217 FinishTime: Mon May 16 14:56:42 CEST 2011 
    218 Number of tasks: 10 
    219 Tasks: calendar_PSit0 calendar_PSit1 calendar_PSit2 calendar_PSit3 calendar_PSit4 calendar_PSit5 calendar_PSit6 calendar_PSit7 calendar_PSit8 calendar_PSit9  
    220 }}} 
    221 {{{#!sh 
    222 $ qcg-client job_info 1305550554579_calendar_example_5366 
    223 UserDN = /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek 
    224 ProxyLifetime = 0 Days 11 Hours 58 Minutes 6 Seconds 
    225  
    226 UserDN: /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek 
    227 Project:  
    228 Status: FINISHED 
    229 StatusDesc:  
    230 SubmissionTime: Mon May 16 14:55:54 CEST 2011 
    231 FinishTime: Mon May 16 14:56:42 CEST 2011 
    232 Number of tasks: 10 
    233 Tasks: calendar_PSit0 calendar_PSit1 calendar_PSit2 calendar_PSit3 calendar_PSit4 calendar_PSit5 calendar_PSit6 calendar_PSit7 calendar_PSit8 calendar_PSit9  
    234 DescriptionType: GRMS 
    235 UserDescription:  
    236 <qcgJob appId="calendar_example"> 
    237   <task persistent="true" taskId="calendar"> 
    238     <execution type="single"> 
    239       <executable> 
    240         <execFile> 
    241           <file> 
    242             <location type="URL">file:////usr/bin/cal</location> 
    243           </file> 
    244         </execFile> 
    245       </executable> 
    246       <arguments> 
    247         <value>${PS_month}</value> 
    248         <value>2010</value> 
    249       </arguments> 
    250       <stdout> 
    251         <file> 
    252           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    253         </file> 
    254       </stdout> 
    255     </execution> 
    256     <parametersSweep> 
    257       <parameter> 
    258         <name>month</name> 
    259         <value> 
    260           <loop> 
    261             <start>1</start> 
    262             <end>12</end> 
    263             <step>1</step> 
    264             <except> 
    265               <value>3</value> 
    266               <value>6</value> 
    267             </except> 
    268           </loop> 
    269         </value> 
    270       </parameter> 
    271     </parametersSweep> 
    272   </task> 
    273 </qcgJob> 
    274  
    275 QCGDescription:  
    276 <grmsJob appId="calendar_example"> 
    277   <task persistent="true" taskId="calendar_PSit0"> 
    278     <execution type="single"> 
    279       <executable> 
    280         <execFile> 
    281           <file> 
    282             <location type="URL">file:////usr/bin/cal</location> 
    283           </file> 
    284         </execFile> 
    285       </executable> 
    286       <arguments> 
    287         <value>1.0</value> 
    288         <value>2010</value> 
    289       </arguments> 
    290       <stdout> 
    291         <file> 
    292           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    293         </file> 
    294       </stdout> 
    295     </execution> 
    296   </task> 
    297   <task persistent="true" taskId="calendar_PSit1"> 
    298     <execution type="single"> 
    299       <executable> 
    300         <execFile> 
    301           <file> 
    302             <location type="URL">file:////usr/bin/cal</location> 
    303           </file> 
    304         </execFile> 
    305       </executable> 
    306       <arguments> 
    307         <value>2.0</value> 
    308         <value>2010</value> 
    309       </arguments> 
    310       <stdout> 
    311         <file> 
    312           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    313         </file> 
    314       </stdout> 
    315     </execution> 
    316   </task> 
    317   <task persistent="true" taskId="calendar_PSit2"> 
    318     <execution type="single"> 
    319       <executable> 
    320         <execFile> 
    321           <file> 
    322             <location type="URL">file:////usr/bin/cal</location> 
    323           </file> 
    324         </execFile> 
    325       </executable> 
    326       <arguments> 
    327         <value>4.0</value> 
    328         <value>2010</value> 
    329       </arguments> 
    330       <stdout> 
    331         <file> 
    332           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    333         </file> 
    334       </stdout> 
    335     </execution> 
    336   </task> 
    337   <task persistent="true" taskId="calendar_PSit3"> 
    338     <execution type="single"> 
    339       <executable> 
    340         <execFile> 
    341           <file> 
    342             <location type="URL">file:////usr/bin/cal</location> 
    343           </file> 
    344         </execFile> 
    345       </executable> 
    346       <arguments> 
    347         <value>5.0</value> 
    348         <value>2010</value> 
    349       </arguments> 
    350       <stdout> 
    351         <file> 
    352           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    353         </file> 
    354       </stdout> 
    355     </execution> 
    356   </task> 
    357   <task persistent="true" taskId="calendar_PSit4"> 
    358     <execution type="single"> 
    359       <executable> 
    360         <execFile> 
    361           <file> 
    362             <location type="URL">file:////usr/bin/cal</location> 
    363           </file> 
    364         </execFile> 
    365       </executable> 
    366       <arguments> 
    367         <value>7.0</value> 
    368         <value>2010</value> 
    369       </arguments> 
    370       <stdout> 
    371         <file> 
    372           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    373         </file> 
    374       </stdout> 
    375     </execution> 
    376   </task> 
    377   <task persistent="true" taskId="calendar_PSit5"> 
    378     <execution type="single"> 
    379       <executable> 
    380         <execFile> 
    381           <file> 
    382             <location type="URL">file:////usr/bin/cal</location> 
    383           </file> 
    384         </execFile> 
    385       </executable> 
    386       <arguments> 
    387         <value>8.0</value> 
    388         <value>2010</value> 
    389       </arguments> 
    390       <stdout> 
    391         <file> 
    392           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    393         </file> 
    394       </stdout> 
    395     </execution> 
    396   </task> 
    397   <task persistent="true" taskId="calendar_PSit6"> 
    398     <execution type="single"> 
    399       <executable> 
    400         <execFile> 
    401           <file> 
    402             <location type="URL">file:////usr/bin/cal</location> 
    403           </file> 
    404         </execFile> 
    405       </executable> 
    406       <arguments> 
    407         <value>9.0</value> 
    408         <value>2010</value> 
    409       </arguments> 
    410       <stdout> 
    411         <file> 
    412           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    413         </file> 
    414       </stdout> 
    415     </execution> 
    416   </task> 
    417   <task persistent="true" taskId="calendar_PSit7"> 
    418     <execution type="single"> 
    419       <executable> 
    420         <execFile> 
    421           <file> 
    422             <location type="URL">file:////usr/bin/cal</location> 
    423           </file> 
    424         </execFile> 
    425       </executable> 
    426       <arguments> 
    427         <value>10.0</value> 
    428         <value>2010</value> 
    429       </arguments> 
    430       <stdout> 
    431         <file> 
    432           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    433         </file> 
    434       </stdout> 
    435     </execution> 
    436   </task> 
    437   <task persistent="true" taskId="calendar_PSit8"> 
    438     <execution type="single"> 
    439       <executable> 
    440         <execFile> 
    441           <file> 
    442             <location type="URL">file:////usr/bin/cal</location> 
    443           </file> 
    444         </execFile> 
    445       </executable> 
    446       <arguments> 
    447         <value>11.0</value> 
    448         <value>2010</value> 
    449       </arguments> 
    450       <stdout> 
    451         <file> 
    452           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    453         </file> 
    454       </stdout> 
    455     </execution> 
    456   </task> 
    457   <task persistent="true" taskId="calendar_PSit9"> 
    458     <execution type="single"> 
    459       <executable> 
    460         <execFile> 
    461           <file> 
    462             <location type="URL">file:////usr/bin/cal</location> 
    463           </file> 
    464         </execFile> 
    465       </executable> 
    466       <arguments> 
    467         <value>12.0</value> 
    468         <value>2010</value> 
    469       </arguments> 
    470       <stdout> 
    471         <file> 
    472           <location type="URL">${TASK_DIR}/stdout.txt</location> 
    473         </file> 
    474       </stdout> 
    475     </execution> 
    476   </task> 
    477 </qcgJob> 
    478 }}} 
    479 }}} 
    480  
    481 === tasks_statuses === 
    482 * `tasks_statuses <jobId>` - lists tasks constituting the given job with their statuses. 
    483 {{{ 
    484 #!div style="font-size: 90%" 
    485 {{{#!sh 
    486 qcg-client tasks_statuses 1305550554579_calendar_example_5366 
    487 UserDN = /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek 
    488 ProxyLifetime = 0 Days 11 Hours 41 Minutes 30 Seconds 
    489  
    490 Number of tasks: 10 
    491 Tasks statuses:  
    492 calendar_PSit7  : FINISHED 
    493 calendar_PSit6  : FINISHED 
    494 calendar_PSit5  : FINISHED 
    495 calendar_PSit4  : FINISHED 
    496 calendar_PSit3  : FINISHED 
    497 calendar_PSit2  : FINISHED 
    498 calendar_PSit1  : FINISHED 
    499 calendar_PSit0  : FINISHED 
    500 calendar_PSit9  : FINISHED 
    501 calendar_PSit8  : FINISHED 
    502 ------ SUMMARY -------- 
    503 Number of tasks: 10 
    504 FINISHED        : 10 
    505 }}} 
    506 }}} 
    507  
    508 === task_info === 
    509 * `task_info <jobId> <taskId>` - displays information about the given job 
    510 {{{ 
    511 #!div style="font-size: 90%" 
    512 {{{#!sh 
    513 qcg-client task_info 1305550554579_calendar_example_5366 calendar_PSit0 
    514 UserDN = /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek 
    515 ProxyLifetime = 0 Days 11 Hours 40 Minutes 11 Seconds 
    516  
    517 TaskType: SINGLE 
    518 SubmissionTime: Mon May 16 14:55:57 CEST 2011 
    519 FinishTime: Mon May 16 14:56:27 CEST 2011 
    520 ProxyLifetime: PT0S 
    521 Status: FINISHED 
    522 StatusDesc:  
    523 StartTime: Mon May 16 14:56:04 CEST 2011 
    524 DescriptionType: <task persistent="true" taskId="calendar_PSit0"> 
    525   <execution type="single"> 
    526     <executable> 
    527       <execFile> 
    528         <file> 
    529           <location type="URL">file:////usr/bin/cal</location> 
    530         </file> 
    531       </execFile> 
    532     </executable> 
    533     <arguments> 
    534       <value>1.0</value> 
    535       <value>2010</value> 
    536     </arguments> 
    537     <stdout> 
    538       <file> 
    539         <location type="URL">${TASK_DIR}/stdout.txt</location> 
    540       </file> 
    541     </stdout> 
    542   </execution> 
    543 </task> 
    544  
    545  
    546 Coallocation:  
    547 UserDN: /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek 
    548 HostName: grass1.man.poznan.pl 
    549 ProcessesCount: 1 
    550 ProcessesGroupId:  
    551 Status: FINISHED 
    552 StatusDescription:  
    553 SubmissionTime: Mon May 16 14:56:04 CEST 2011 
    554 FinishTime: Mon May 16 14:56:21 CEST 2011 
    555 LocalSubmissionTime: Mon May 16 14:56:06 CEST 2011 
    556 LocalStartTime: Mon May 16 14:56:10 CEST 2011 
    557 LocalFinishTime: Mon May 16 14:56:10 CEST 2011 
    558 }}} 
    559 }}} 
     48[qcg-advance-client]