[[PageOutline]]

= application =
'''application''' - name and optionally version of the application to be executed
{{{
#QCG application=namd
}}}
{{{
#QCG application=gromacs/4.6.3
}}}


= argument =
'''argument''' - argument of the application. The directive can be used many times, once for every argument.
{{{
#QCG argument=arg1
#QCG argument=arg2
}}}


= assistant =
'''assistant''' - starts in parallel to the main application the auxiliary script that will be terminated automatically with the end of the task.
The script can be used for example to monitor output of the application or to do perform any additional action. The directive was introduced in the 3.4 version and replaced the ''monitoring'' directive which meaning has changed.

Syntax:
{{{
#QCG assistant=script.sh
#QCG assistant=script->script.sh
}}}

Parameters:
* script - mandatory, default parameter defining the script to be executed. The file must be transfered using the #QCG stage-in-file directive.
{{{
#QCG assistant=script.sh
#QCG stage-in-file=script.sh
}}}

'''IMPORTANT:''' The script is executed once at the beginning of the main application. If any action is expected to be executed periodically it must be programmed inside the script by the user.

= deadline = 
'''deadline''' - deadline for the task execution. 
The directive specifies, that the task has to be finished within defined period of time. 

The directive results in using the advance reservation to guarantee the requested quality of service. 

The directive cannot be used in conjunction with the ''reservation'' and ''not-after'' directives. 

The argument of the directive is the time period defined in the PnYnMnDTnHnMnS (ISO 8601) format. For details please see the '''walltime''' directive.

{{{
#QCG deadline=P3DT12H
#QCG not-before=2012.07.25
}}}
{{{
#QCG deadline=P3DT12H
}}}

= environment =
'''environment''' - environment variable. The syntax is ''key'' -> ''value''. Every variable has be be specified in separate directive.

{{{
#QCG environment=name -> piontek
#QCG environment=location -> poznan
}}}

= error =
'''error''' - location where the system is requested to stage out the file containing output of the standard error stream. If the location is not the gsiftp:// url it is assumed that it is relative to the directory from which the task was submitted.

{{{
#QCG error=gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/errors/${JOB_ID}.error
#QCG error=error.txt
}}}


= executable =
'''executable''' - location of the executable file. The location can be the gsiftp:// url or the path to the file. In the second case the location is relative to the directory from which the task was submitted. Optionally it is possible to specify the target name of the file.
{{{
#QCG executable=gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/executables/exec1
#QCG executable=executables/exec1
#QCG executable=executables/exec1 -> exec-file
}}}
= grant = 
'''grant''' - name of the grant, in scope of which the task is executed

{{{
#QCG grant=plgpiontek_grant
}}}

= host =
'''host''' - name of the candidate host, which the task/reservation can be executed/made on. The directive can appear many times to express OR alternative.

{{{
#QCG host=reef.man.poznan.pl
#QCG host=zeus.cyfronet.pl
}}}

= include =
'''include''' - directive allowing to include file into description. Supported are relative and absolute paths.
{{{
#QCG include=includes/application.qcg
#QCG include=/home/piontek/includes/application.qcg
}}}

= input =
'''input''' - location of the file which content will be passed to the standard input stream. If the location is not the gsiftp:// url it is assumed that it is relative to the directory from which the task was submitted. 

{{{
#QCG input=gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/inputs/experiment.input
#QCG input=input.txt
}}}

= memory =
'''memory''' - declaration of the maximal amount of memory that can be allocated by the application (in MB).

{{{
#QCG memory=1024
}}}

= memory-per-slot =
'''memory-per-slot''' - declared maximal amount of the memory per slot (core). The total memory requirement is the product of this value and the number of slots (cores) used by the application.

{{{
#QCG memory-per-slot=512
}}}

= module =
'''module''' - name with optional version of the requested software module. This directive is taken into consideration by the broker in the process of choosing the optimal resource which the task can be executed on. Every module has to be specified in separate directive.

{{{
#QCG module=nwchem/6.0
#QCG module=namd
}}}

= monitor =
'''monitor''' - the directive allowing to monitor the progress and status of the application in the QCG-Monitoring service.

'''IMPORTANT:" The meaning of the ''monitoring'' directive has changed in the version 3.4 of the QCG-SimpleClient. The previous functionality is available by using the #QCG assistant directive.


Syntax:
{{{
#QCG monitor[=script]
#QCG monitor=script->script.sh[,delay->N]
}}}

For the applications having only one monitoring schema the directive can be parameterless.
{{{
#QCG monitor
}}}

To define the monitoring schema or the script implementing the monitoring functionality, the ''script'' parameter is dedicated.
The ''script'' parameter is the default one.
Providing its own monitoring script requires it to be transfered using the #QCG stage-in-file directive.

{{{
#QCG monitor=gaussian
 
#QCG monitor=skrypt_monitorujący.sh
#QCG stage-in-file=skrypt_monitorujący.sh
}}}

The frequency of script execution is defined by the ''delay'' parameter which value defined time is seconds between next executions of the script.
The notification is sent only when a content of the notification has changed from the previous iteration.
{{{
#QCG monitor=gaussian,delay->60
 
#QCG monitor=script->gaussian,delay->60
}}}
 
=mount=
* '''mount''' - the directive automatically mounts for the task the defined data management system. At the moment only the ''veilFS'' system is supported (PL-Grid specific).

{{{
#QCG mount=veilfs[->directive]
}}}

The optional ''directive'' argument defines the location in which the system is mounted. The default value is ''mount''.

{{{
#QCG mount=veilfs->veilFS_dir
}}}
= name =
'''name''' - human readable name of the task. The name will appear as part of the job identifier.

{{{
#QCG name=nobel-experiment
}}}
= native =
'''native''' - parameters/requirements passed directly to the queueing system in the queueing system specific format. This directive should be used together with the ''host'' one.

Example for the PBS queueing system:
(-r y|n Declares whether the job is rerunable.)

{{{
#QCG native=-r n
}}}

= nodes =
'''nodes''' - requested topology of the parallel application. The directive allows to specify requested number of nodes and slots per node. Optionally it is possible to specify number of processes to be started on every node. If it is not specified else the number of processes on every node is equal to the number of slots.
The syntax is: number_of_nodes:number_of_slots[:number_of_processes]

{{{
#QCG nodes=10:5:1
#QCG nodes=12:12
}}}

= not-after =
'''not-after''' - directive allowing to specify that the task or reservation has to be executed/reserved "not-later-then". The value of this directive is date with optional time. The directive requires to be specified together with the ''walltime'' one and cannot be used with the ''reservation'' one.

{{{
#QCG not-after=2012.08.25
#QCG walltime=PT2H
}}}

{{{
#QCG not-after=2012.08.25 13:20
#QCG walltime=PT2H
}}}

= not-before =
not-before - directive allowing to specify that the task or reservation has to be executed/reserved "not-earlier-then". The value of this directive is the date with optional time. The directive requires to be specified together with the ''walltime'' one and cannot be used with the ''reservation'' one.

{{{
#QCG not-before=2012.07.25
#QCG walltime=PT2H
}}}
{{{
#QCG not-before=2012.07.25 15:20
#QCG walltime=PT2H
}}} 

= note =
'''note''' - human readable description of the task

{{{
#QCG note=my first QCG task
}}}

= notify =
'''notify''' - notify about changes of task status. The directive allows to define the protocol and address to which the notification about the change of the status should be sent. Supported protocols are mail and xmpp/jabber messages.

{{{
#QCG notify=mailto:piontek@man.poznan.pl
#QCG notify=xmpp:tomasz.piontek@gmail.com
}}}

To allow receiving xmpp/jabber messages it can be necessary to add the qcg-notification@plgrid.pl sender address to contacts.

= output =
'''output''' - location where the file containing the standard output should be transfered. The location can be the gsiftp:// url or the path relative to the directory from which the task was submitted.

{{{
#QCG output=gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/outputs/${JOB_ID}.output
#QCG output=output.txt
}}}

= parameter-sweep =

'''parameter-sweep''' - allows to define name of the variable and a space of the parameter values for this variable. The directive can be used many times defining multi-dimensional space of parameters. For every parameters' combination a separate task is launched. 

Syntax:
{{{
#QCG parameter-sweep=VAR->PARAMETER_SPACE
}}}

The following types of definition are supported:

* ''list''' - list of comma-separated values
{{{
#QCG parameter-sweep=var->list:mon,tue,wed,thu,fri,sat,sun
#QCG parameter-sweep=var->list:1,2,3,4,5,6
}}}

* ''for''' - values change from ''start'' upto ''end'' with the ''step'' step. Default step is 1. Number decimal digits of the value is equal to the maximal number of decimal digits of ''start'', ''end'', ''step'' values.

Syntax:
{{{
#QCG parameter-sweep=var->for:start..end
#QCG parameter-sweep=var->for:start..end..step
}}}

{{{ 
#QCG parameter-sweep=var->for:0..10
#QCG parameter-sweep=var->for:0..10..2
#QCG parameter-sweep=var->for:1.0..2.0..0.5
}}}

* ''pattern'' - values are names of the files matching the given pattern.
{{{
#QCG parameter-sweep=var->pattern->*.input
}}}

* ''file'' - values are taken from next lines of the given file.
{{{
#QCG parameter-sweep=var->file:file_with_values.txt
}}}
= persistent =
'''persistent''' - the directive informing the system to not remove the working directory. 

'''IMPORTANT:''' the directive should be used only for testing as it causes that the working directory is left and consumes storage resource. The working directory can be later cleaned by the '''qcg-clean''' command.

{{{
#QCG persistent
}}}

= postprocess =
'''postprocess''' - task epilog. The value of the directive can be command or path to the script which is executed when the main task finish.

{{{
#QCG postprocess=tar cvf wynik.tar *
}}}
{{{
#QCG postprocess=postprocess-script.sh
}}}

= preprocess =
'''preprocess''' - task prolog. The value of the directive can be command or path to the script which is executed before the main task.

{{{
#QCG preprocess=mkdir outputs
}}}
{{{
#QCG preprocess=preprocess-script.sh
}}} 

= procs =
'''procs''' - number of requested slots/processes for parallel tasks.

{{{
#QCG procs=32
}}}

= properties =
'''properties''' - node properties. Defines requested properties of the node which the task can be executed on.

{{{
#QCG properties=mpi,ib,lustre
}}}

= queue =
'''queue''' - queue which the task should be submitted to.

{{{
#QCG queue=plgrid
}}}

= reservation =
'''reservation''' - reservation identifier. The identifier can be of the '''local''' type when the reservation was made directly in the queueing system or of the '''qcg''' one (default) when the reservation was created by the qcg-reserve command. The directive requires to be used together with the '''host''' one to ensure that the task will be submitted to the host where the reservation was done.

{{{
#QCG host=reef.man.poznan.pl
#QCG reservation=R1366398248299__4039
}}}

{{{
#QCG host=reef.man.poznan.pl
#QCG reservation=piontek.0 local
}}}

= stage-in-dir =
'''stage-in-dir''' - input directory. The location of the input directory which can be either the gsiftp:// url or the path to the directory relative to the directory from which the task was submitted. The syntax is "location" -> "target name". If the "target name" is not present the directory is stored under its original name.

{{{
#QCG stage-in-dir=gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/inputs -> inputs
#QCG stage-in-dir=gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/inputs
#QCG stage-in-dir=input_dir -> inputs
#QCG stage-in-dir=input_dir
}}}

To stage in the whole directory which the task was submitted from please use the "." (dot) as a location.

{{{
#QCG stage-in-dir=. -> .
#QCG stage-in-dir=.
#QCG stage-in-dir=. -> input
}}}

= stage-in-file =
'''stage-in-file''' - input file. The location of the input file which can be either the gsiftp:// url or the path to the directory relative to the directory which the task was submitted from. The syntax is "location" -> "target name". If the "target name" is not present the file is stored under its original name. 

{{{
#QCG stage-in-file=gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/inputs/input.txt -> input.txt
#QCG stage-in-file=gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/inputs/input.txt
#QCG stage-in-file=input_file.txt -> input.txt
#QCG stage-in-file=input_file.txt
}}}

= stage-out-dir =
'''stage-out-dir''' - output directory. The location of the output directory which can be either the gsiftp:// url or the path to the directory relative to the directory which the task was submitted from. The syntax is "location" -> "target name". If the "target name" is not present the directory is stored under its original name. 

{{{
#QCG stage-out-dir=results -> gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/results/${JOB_ID}
#QCG stage-out-dir=results -> gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/results/${JOB_ID}
#QCG stage-out-dir=results -> result
#QCG stage-out-dir=results
}}}

To transfer the whole working directory please specify the location as "." (dot).

{{{
#QCG stage-out-dir=. -> .
#QCG stage-out-dir=.
#QCG stage-out-dir=. -> output
}}}

= stage-out-file =
'''stage-out-file''' - output file

The location of the output file which can be either the gsiftp:// url or the path to the file relative to the directory which the task was submitted from. The syntax is "location" -> "target name". If the "target name" is not present the file is stored under its original name. 

{{{
#QCG stage-out-file=results.txt -> gsiftp://qcg.man.poznan.pl//home/plgrid/plgpiontek/reef/results/result.1
#QCG stage-out-file=results.txt
#QCG stage-out-file=result.txt -> ${JOB_ID}.result
#QCG stage-out-file=result.txt
}}}

= use-reservation =
'''use-reservation''' - use the advance reservation mechanism. The directive allows to force the system to use advance reservation in case when directives '''not-after''', '''not-before''', '''deadline''' were not used. The directive requires to be used together with the '''walltime''' one. The directive cannot be used together with the '''reservation''' one. 

{{{
#QCG use-reservation
#QCG walltime=PT2H
}}}
= use-scratch =
'''use-scratch''' - use the scratch space. The directive forces the system to use the scratch space if possible.

{{{
#QCG use-scratch
}}}

= walltime =
'''walltime''' - execution duration. The directive specifies declared by the user duration of the task execution or resources reservation.

The argument of the directive is the time period defined in the PnYnMnDTnHnMnS (ISO 8601) format, where:
* P - mandatory character,
* nY - number of years,
* nM - number of months,
* nD - number of days,
* T - time definition separator character (must be present if the values below are used)
* nH - number of hours,
* nM - number of minutes,
* nS - number of seconds.

{{{
#QCG walltime=P3DT12H
}}}

= watch-output =

* '''watch-output''' - dirrective allowing to monitor the application output and sending notification messages.
   
'''IMPORTANT:''' - starting frome the QCG 3.4 version the syntax of the directive has changed.

Syntax:
{{{
#QCG watch-output=consumer->consumer_url,[pattern->pattern_value | script->script_name ,delay->N]
#QCG watch-output=consumer_url
}}}

Parameters:
* ''consumer'' - mandatory, default parameter specifying the notification consummer. (supported protocols are: mail i xmpp - see: notify directive)
* ''pattern'' or ''script'' - parameter specifying either the pattern (regular expression) or the whole monitoring script. In both cases the output is sent to the consumer. In the case of the script it has to be transfered using the #QCG stage-in-file directive.
* ''delay'' - time in seconds between next invocations of the monitoring procedure. The message is sent only when the output has changed from the previous interation.

{{{
#QCG watch-output=mailto:piontek@man.poznan.pl,pattern->ENERGY,dalay->60
 
#QCG watch-output=consumer->mailto:piontek@man.poznan.pl,script->skrypt_monitorujacy.sh,dalay->60
#QCG stage-in-file=skrypt_monitorujacy.sh
}}}
= variable =
'''variable''' - description variable (replaced only in values of QCG directives) 

{{{
#QCG variable=PROCESSES->10
#QCG procs=${PROCESSES}
}}}
'''variable''' - description variable (replaced only in values of QCG directives)