Changes between Version 99 and Version 100 of benchmarks

Timestamp:

11/24/11 11:08:34 (12 years ago)

Author:

bartek

Comment:

--

benchmarks

@@ -1 +1 @@
 = Benchmarks of !QosCosGrid =
 == QCG-Computing ==
-The QCG-Computing service tests concerned the job submission and job management tasks, which are typical for this kind of a service. In the tests QCG-Computing was compared to gLite and UNICORE. The proposed two types of the tests used the following metrics:
-* response time,
-* throughput 
+[[http://www.qoscosgrid.org/trac/qcg-computing/wiki/benchmarks]]
+== QCG-Notification ==
+[[http://www.qoscosgrid.org/trac/qcg-notification/wiki/benchmarks]]
 
-All the tests were performed using specially written program on the basis of the SAGA C++ library. There were utilized two adaptors offered by SAGA C++, namely:
-* gLite CREAM (based on glite-ce-cream-client-api-c) - gLite (CREAM-CE service),
-* OGSA BES (based on gSOAP) - UNICORE and !QosCosGrid (QCG-Computing service).
-
-The use of the common access layer allowed to minimize the risk of obtaining incorrect results. In the same purpose, the jobs were submitted to the same resource and didn't require any data transfer. 
-
-
-=== Testbed ===
-* Client machine:
- * 8 cores (Intel(R) Xeon(R) CPU E5345),
- * 11 GB RAM,
- * Scientific Linux 5.3,
- * RTT from the client's machine to the cluster's frontend: about 12 ms.
-* Cluster Zeus (84. place on TOP500):
- * queueing system: Torque 2.4.12 + Maui 3.3,
- * about 800 nodes,
- * about 3-4k tasks present in the system,
- * Maui „RMPOLLINTERVAL”: 3,5 minutes,
- * for the puropose of the tests, a special partition (WP4) was set aside: 64 cores / 8 nodes - 64 slots,
- * test users (plgtestm01-10 and 10 users from the plgridXXX pool) were assigned on an exclusive basis to the WP4 partition.
-* Service nodes (qcg.grid.cyf-kr.edu.pl, cream.grid.cyf-kr.edu.pl, uni-ce.grid.cyf-kr.edu.pl):
- * virtual machines (Scientific Linux 5.5),
- * QCG and UNICORE: 1 virtual core, 2GB RAM,
- * gLite CREAM: 3 virtual cores, 8 GB RAM.
-
-=== Test 1 - Response Time ===
-The main program creates N processes (each process can use a different certificate) that invoke the function `sustain_thread`. Next, it waits for the end of all running processes.
-
-In general, the idea of the program is to keep in a system `jobs_per_thread` jobs for `test_duration` seconds, inquering all the time (the delays between calls drawn from a defined interval) about all currently running or queued jobs.
-
-The following snippet shows a pseudocode of the function `sustain_thread`:
-{{{
-#!div style="font-size: 90%"
-{{{#!sh
-1. start_timer()
-2. for i = 1 .. jobs_per_thread
-  2a: submit_job(job[i])
-3. while (current_time < test_duration) do
-  3a: for i = 1 .. jobs_per_thread
-  3a1: if (! is_finished(job[i].last_state))
-    3a11: sleep((rand() / RAND_MAX) / SLEEP_COEF)
-    3a11: query_state(job[i])
-  3a2: if (is_finished(job[i].last_state))
-    3a21: submit_job(job[i])
-4. stop_timer()
-}}}
-}}}
-The function `submit_job(job)`:
-{{{
-#!div style="font-size: 90%"
-{{{#!sh
-1. start_timer()
-2. job.job = service.create_job()
-3. job.job.run()
-4. stop_timer()
-5. query_state(job)
-}}}
-}}}
-The function `query_state(job)`:
-{{{
-#!div style="font-size: 90%"
-{{{#!sh
-1. start_timer()
-2. job.last_state = job.job.get_state()
-3. stop_timer()
-}}}
-}}}
-
-At the end of tests, the average, minimal and maximal times of submitting a job (`submit_job`) and inquiring about a job state (`query_state`) are printed. Additionally, the program displays the number of all submitted jobs, the number of successfully finished jobs (`Done`) and the number of the jobs finished with the other status (`Canceled`, `Failed`, `Suspended`). In the last case, the number of fails, i.e. exceptions returned by the SAGA adaptors, is shown. 
-
-==== Notes ====
-
-''Pros:''
-* The test reflects the natural situation in productive environments:
- * approximately constant number of tasks,
- * "the task flow" (when one task is finished, another begins).
-* The program may be used to measure the overall capacity of the system.
-
-''Cons:''
-* The measured submitting time may be distorted (the response of the service on the submit request does not necessarily imply the submit to the queueing system).
-
-==== Plan of the tests ====
-* 50 tasks x 10 users = 500 tasks, 30 minutes, SLEEP_COEF = 10
-* 100 tasks x 10 users = 1000 tasks, 30 minutes, SLEEP_COEF = 10
-* 200 tasks x 10 users = 2000 tasks, 30 minutes, SLEEP_COEF = 10
-* 400 tasks x 10 users = 4000 tasks, 30 minutes, SLEEP_COEF = 10
-
-==== Results ====
-* Average submit time of a single job
- 
- || ||= QCG 2.0 =||= UNICORE 6.3.2 =||= gLite 3.2 =||
- ||= 50x10 =||1.43||2.41||8.47||
- ||= 100x10 =||1.49||1.24^[#n 1]^||8.45||
- ||= 200x10 =||1.99||2.20||8.50||
- ||= 400x10 =||1.96||-^[#n 2]^||8.24||
-
-
-* Average time of a query about a job status.
- 
- || ||= QCG 2.0 =||= UNICORE 6.3.2 =||= gLite 3.2 =||
- ||= 50x10 =||0.38||2.73||0.20||
- ||= 100x10 =||0.35||1.61^[#n 1]^||0.36||
- ||= 200x10 =||0.63||3.73||0.24||
- ||= 400x10 =||0.47||-^[#n 2]^||0.21||
-
-* The system load while performing the test "200x10" (on a basis of Ganglia)
-[[Image(ganglia-200x10-all.png, center, width=880px)]]
-
-=== Test 2 - Throughput ===
-The test is grounded on the methodology described in the paper [[http://dl.acm.org/citation.cfm?id=1533455|Benchmarking of Integrated OGSA-BES with the Grid Middleware]] and bases on measurement performed from the user perspective of the finish time of the last from N jobs submitted at (almost) the same moment. In addition to the paper, the presented test has utilized also the following elements: 
-* submitting the tasks by N processes/users,
-* using consistent SDK, not the command-line clients,
-* single test environment.
-
-==== Notes ====
-
-''Pros:''
-* Possibility to compare results of the tests with the results presented in the mentioned publication. 
-* Measures the performance of the system from a user point of view (from the first job submit to the finish of the last job).
-
-''Cons:''
-* The test is valid only if jobs are submitted to a separated resource.
-* The tasks of a scheduler of a system (Maui) may have significant impact on results (the scheduler performs complex operations from time to time).
-
-==== Results ====
-* 1 user, 1 thread, 500 tasks:
-[[Image(zeus-throughput-500x1-1.png, center, width=640px)]]
-* 1 user, 10 thread, 500 tasks (50x10):
-[[Image(zeus-throughput-50x1-1.png, center, width=640px)]]
-* 10 users, 10 thread, 500 tasks (50x10):
-[[Image(zeus-throughput-50x10-0.png, center, width=640px)]]
-* 10 users, 10 thread, 1000 tasks (10x100):
-[[Image(zeus-throughput-100-10-0.png, center, width=640px)]]
-
-=== Notes ===
-1. The machine where CREAM (gLite) was running had more resources (in particular CPU cores and virtual memory) than the machines with QCG and UNICORE.
-2. ... hovewer this machine was additionally loaded by external tasks (about 500-2000 tasks - the tests were performed by 2 weeks).
-3. QCG returns the job status when the job is already in queueing system, gLite and UNICORE not necessarily. Thus, e.g. in the throughput tests, new tasks appeared after the test finished.
-4. The bottle-neck (especially in the second group of tests) was the throughput of the WP4 partition and Maui, which imposed that only 64 tasks could be scheduled per one scheduling cycle (at least 3.5 minutes).
-
-
-== QCG-Notification ==
-
-The QCG-Notification performance was compared to the performance of two other services implementing brokered version of WS-Notification specification. These were Apache !ServiceMix (version 3.3.1) and IBM !WebSphere Application Server (version 7.0). 
-
-A reliable performance tests of the notification systems are difficult to realise. Note that the proceeding time of a single, small message is highly short, difficult to correct measurement and highly depends on many aspects, such as machine configuration, underlying best effort transport protocols or network protocols. To overcome this problem, the time was measured not on the basis of a single send of a notification, but on the basis of multiple sends (called here bunch) assumed as an elementary event.
-
-Taking into account these objectives as well as the fact that the tested notification systems are different: each system provides a set of specific functions that may be unavilable in the another systems (for example, !ServiceMix doesn’t provide support for Full topic dialect), the decision was to test only the most common features and focus on key scenarios.
- 
-The elementary events were performed many times and the received results were averaged.
-
-=== Testbed ===
-The testbed consisted of two homogenous machines connected over the 1Gb Ethernet (see the attached figure). The first machine was used to host the lightweight versions of notification systems clients, that are Publisher and !NotificationConsumer, while the second machine was dedicated for the three tested notification systems and additionally for the another instance of !NotificationConsumer to allow simple tests.
-
-[[Image(tests-deployment.png, center, width=540px)]]
-
-=== Test 1 - Notification Delivery without Active Subscriptions ===
-The first test concerned the measurement of the time of sending a bunch of notifications (hovewer each notification is sent separately) while there were no active subscriptions in the tested system. That means that the time was counted on the Publisher side and the notifications were not forwarded by the brokers as there were any Notification Consumers interested in receiving notifications. 
-
-==== Main facts ====
-* Scenario's draft: Publisher -> Notification Broker
-* Time measurement: on Publisher's side
-* Filtering based on: topics
-* Notification dialect: Simple
-* Notification payload: about 320 bytes
-* Number of notifications in one bunch: 1000
-
-==== Notes ====
-
-''Pros:''
-* Possibility to compare performance of the systems in the most basic, but typical scenario, where notifications are not forwarded to Notification Consumers.
-* The delay imposed by the notification systems as well as the network may be easily measured by comparison to the times of sending notifications between lightweight Publisher and Notification Consumer programs.
-
-''Cons:''
-* The Publisher installed only on a single machine.
-* The notification services in a default configuration and only marginally loaded (no subscriptions).
-* The measurement of time on the Publisher site may be unreliable because the service can schedule the processing of notifications for future.
-==== Results ====
-
-[[Image(test-1.png, center, width=540px)]]
-
-=== Test 2 - Notification Delivery with Active Subscriptions ===
-In this test the measured value was the time of forwarding notifications from Publisher to Notification Consumer via different systems. In contrast to the previous test, here the tested systems utilized Subscriptions and on their basis determined where to send a certain notification. The next importand difference was the move of the finish time measurement from Publisher to Notification Consumer.
-
-==== Main facts ====
-* Scenario's draft: Publisher -> Notification Broker -> Notification Consumer
-* Time measurement: on Publisher's side (start time) and on Notification Consumer's side (finish time)
-* Filtering based on: topics
-* Notification dialect: Simple
-* Notification payload: about 320 bytes
-* Number of notifications in one bunch: 1000
-
-==== Notes ====
-
-''Pros:''
-* Possibility to compare performance of the systems in the typical scenario that takes into account notification forwarding to Notification Consumers.
-* The time is measured not only by Publisher but also by Notification Consumer which gives more precise results of processing time.
-
-''Cons:''
-* The Publisher and Notification Consumer programs installed only on a single, the same machine.
-* The notification services in a default configuration and marginally loaded (only several subscriptions).
-* !WebSphere Application Server concatenates received messages if they are forwarded to the same Notification Consumer - although in general it improves the performance, sometimes it may impose delays in sending to consumers and improper results. Nevertheless this point was practically neutralized thanks to measuring the time of large amount of elementary operations.
-
-==== Results ====
-[[Image(test-2.png, center, width=340px)]]
-
-
-=== Test 3 - Subscribing ===
-The aim of the third benchmark was the comparison of the times of creating subscriptions in the tested systems. 
-In the Subscribing test, a lightweight Subscriber program created the same number of the same subscriptions in each system. Similarly to the previous tests, the time, which was measured by the Subscriber program, covered a bunch of invokes of a single Subscribe operation.
-
-==== Main facts ====
-* Scenario's draft: Subscriber -> Notification Broker
-* Time measurement: by Subscriber
-* Filtering based on: topics
-* Subscription dialect: Simple
-* !SubscribeRequest payload: about 300 bytes
-* !SubscribeResponse payload: about 280 bytes
-* Number of Subscribe operations in one bunch: 100
-
-==== Notes ====
-
-''Pros:''
-* Possibility to compare the time of creating common subscriptions in the tested systems.
-
-''Cons:''
-* Only one Subscriber program.
-* The notification services in a default configuration; no other operations invoked while subscribing - poorly realistic situation.
-
-==== Results ====
-[[Image(test-3.png, center, width=340px)]]
-
-
-
-[=#n]
-----[[BR]]
-^1^ The test performed after the restart of the machine - caused by mulfunctioning of the LUSTRE system.[[BR]]
-^2^ Error: "End of file or no input: Resource temporarily unavailable".[[BR]]