QCG-Computing Benchmarks

The QCG-Computing service benchmarking tests concerned the job submission and job management tasks, which are typical for this kind of a service. In the tests QCG-Computing was compared to gLite and UNICORE. The proposed two types of the tests used the following metrics:

  • response time,
  • throughput.

All the tests were performed using specially written program on the basis of the SAGA C++ library. There were utilized two adaptors offered by SAGA C++, namely:

  • gLite CREAM (based on glite-ce-cream-client-api-c) - gLite (CREAM-CE service),
  • OGSA BES (based on gSOAP) - UNICORE and QosCosGrid (QCG-Computing service).

The use of the common access layer allowed to minimize the risk of obtaining incorrect results. In the same purpose, the jobs were submitted to the same resource and didn't require any data transfer.


  • Client machine:
    • 8 cores (Intel(R) Xeon(R) CPU E5345),
    • 11 GB RAM,
    • Scientific Linux 5.3,
    • RTT from the client's machine to the cluster's frontend: about 12 ms.
  • Cluster Zeus (84. place on TOP500):
    • queueing system: Torque 2.4.12 + Maui 3.3,
    • about 800 nodes,
    • about 3-4k jobs present in the system,
    • Maui „RMPOLLINTERVAL”: 3,5 minutes,
    • for the puropose of the tests, a special partition (WP4) was set aside: 64 cores / 8 nodes - 64 slots,
    • test users (plgtestm01-10 and 10 users from the plgridXXX pool) were assigned on an exclusive basis to the WP4 partition.
  • Service nodes (qcg.grid.cyf-kr.edu.pl, cream.grid.cyf-kr.edu.pl, uni-ce.grid.cyf-kr.edu.pl):
    • virtual machines (Scientific Linux 5.5),
    • QCG and UNICORE: 1 virtual core, 2GB RAM,
    • gLite CREAM: 3 virtual cores, 8 GB RAM.

Test 1 - Response Time

The main program creates N processes (each process can use a different certificate) that invoke the function sustain_thread. Next, it waits for the end of all running processes.

In general, the idea of the program is to keep in a system jobs_per_thread jobs for test_duration seconds, inquering all the time (the delays between calls drawn from a defined interval) about all currently running or queued jobs.

The following snippet shows a pseudocode of the function sustain_thread:

1. start_timer()
2. for i = 1 .. jobs_per_thread
  2a: submit_job(job[i])
3. while (current_time < test_duration) do
  3a: for i = 1 .. jobs_per_thread
  3a1: if (! is_finished(job[i].last_state))
    3a11: sleep((rand() / RAND_MAX) / SLEEP_COEF)
    3a11: query_state(job[i])
  3a2: if (is_finished(job[i].last_state))
    3a21: submit_job(job[i])
4. stop_timer()

The function submit_job(job):

1. start_timer()
2. job.job = service.create_job()
3. job.job.run()
4. stop_timer()
5. query_state(job)

The function query_state(job):

1. start_timer()
2. job.last_state = job.job.get_state()
3. stop_timer()

At the end of tests, the average, minimal and maximal times of submitting a job (submit_job) and inquiring about a job state (query_state) are printed. Additionally, the program displays the number of all submitted jobs, the number of successfully finished jobs (Done) and the number of the jobs finished with the other status (Canceled, Failed, Suspended). In the last case, the number of fails, i.e. exceptions returned by the SAGA adaptors, is shown.



  • The test reflects the natural situation in productive environments:
    • approximately constant number of jobs,
    • "the jobs flow" (when one job is finished, another begins).
  • The program may be used to measure the overall capacity of the system.


  • The measured submitting time may be distorted (the response of the service on the submit request does not necessarily imply the submit to the queueing system).

Plan of the tests

  • 50 jobs x 10 users = 500 jobs, 30 minutes, SLEEP_COEF = 10
  • 100 jobs x 10 users = 1000 jobs, 30 minutes, SLEEP_COEF = 10
  • 200 jobs x 10 users = 2000 jobs, 30 minutes, SLEEP_COEF = 10
  • 400 jobs x 10 users = 4000 jobs, 30 minutes, SLEEP_COEF = 10


  • Average submit time of a single job

QCG 2.0 UNICORE 6.3.2 gLite 3.2
50x10 1.432.418.47
100x10 1.491.2418.45
200x10 1.992.208.50
400x10 1.96-28.24
  • Average time of a query about a job status.

QCG 2.0 UNICORE 6.3.2 gLite 3.2
50x10 0.382.730.20
100x10 0.351.6110.36
200x10 0.633.730.24
400x10 0.47-20.21
  • The system load while performing the test "200x10" (on a basis of Ganglia)

Test 2 - Throughput

The test is grounded on the methodology described in the paper  Benchmarking of Integrated OGSA-BES with the Grid Middleware and bases on measurement performed from the user perspective of the finish time of the last from N jobs submitted at (almost) the same moment. In addition to the paper, the presented test has utilized also the following elements:

  • submitting the jobs by N processes/users,
  • using consistent SDK, not the command-line clients,
  • single test environment.



  • Possibility to compare results of the tests with the results presented in the mentioned publication.
  • Measures the performance of the system from a user point of view (from the first job submit to the finish of the last job).


  • The test is valid only if jobs are submitted to a separated resource.
  • The tasks of a scheduler of a system (Maui) may have significant impact on results (the scheduler performs complex operations from time to time).


  • 1 user, 1 thread, 500 jobs:

  • 1 user, 10 thread, 500 jobs (50x10):

  • 10 users, 10 thread, 500 jobs (50x10):

  • 10 users, 10 thread, 1000 jobs (10x100):


  1. The machine where CREAM (gLite) was running had more resources (in particular CPU cores and virtual memory) than the machines with QCG and UNICORE.
  2. ... hovewer this machine was additionally loaded by external jobs (about 500-2000 jobs - the tests were performed by 2 weeks).
  3. QCG returns the job status when the job is already in queueing system, gLite and UNICORE not necessarily. Thus, e.g. in the throughput tests, new jobs appeared after the test finished.
  4. The bottle-neck (especially in the second group of tests) was the throughput of the WP4 partition and Maui, which imposed that only 64 jobs could be scheduled per one scheduling cycle (at least 3.5 minutes).

1 The test performed after the restart of the machine - caused by mulfunctioning of the LUSTRE system.
2 Error: "End of file or no input: Resource temporarily unavailable".