Version 19 (modified by mmamonski, 8 years ago) (diff)

--

Application Developer Information

The main goal of the QosCosGrid middleware was a construction of a flexible, efficient and secure distributed IT system being able to deal with large-scale simulations onto distributed computing resources connected over local and wide area networks, in particular using Internet connections. From the development perspective, QosCosGrid supports three classes of use cases covering a wide-range of possible applications, namely: ANSI C or similar use cases, which rely on the Message Passing Paradigm, Java use cases taking the advantage of ProActive library as the parallelization technology and multi-scale use cases based on MUSCLE library.

QosCosGrid services give a great opportunity for administrators to share and use their clusters more efficiently by scheduling local and Grid end users applications according to defined policies and constraints. Advanced scheduling, monitoring and accounting features provided by QosCosGrid improve the utilization of computing resources and throughput. All end users activities in local cluster can be easily checked or reported on request using administrative and external simulation tools.

QCG OpenMPI

The Message Passing Interface (MPI) is de facto a standard in the domain of parallel applications demanding computational resources that are beyond what single machine can provide. It delivers end-users both the programming interface consisting of simple communication primitives and the environment for spawning and monitoring MPI processes. A variety of implementations of the MPI standard is available (both as commercial and open source). In QosCosGrid, it was decided to use OpenMPI implementation of the MPI 2.0 standard as input for further enhancements. Of key importance were the inter-cluster communication techniques that deal with firewalls and Network Address Translation. In addition, the mechanism for spawning new processes in OpenMPI needed to be integrated with QosCosGrid-developed middleware. The extended version of the OpenMPI framework was named  QCG-OMPI (where QCG stands for QosCosGrid). The extensions were three-fold:

  1. internally, QCG-OMPI improves the MPI library by featuring multiple connectivity techniques to enable, when possible, direct connections between MPI ranks that are located in remote clusters potentially separated by firewalls;
  2. the MPI standard was extended to comply with the QosCosGrid semi-opportunistic approach, by providing a new interface to describe the actual topology provided by the meta-scheduler;
  3. many MPI collective operations were upgraded to be hierarchy-aware, and optimized for the Grid.

MUSCLE support

Multi-scale simulations often require a large computation resources. Within the Mapper project, QosCosGrid was extended and adjusted to support cross-cluster execution of applications based on a MUSCLE library. For more information about Mapper go to  http://www.mapper-project.eu/

Cross-cluster communication

See QCG Libraries and Cross-cluster communication.

Eclipse Parallel Tools Platform (PTP) for QosCosGrid

The Parallel Tools Platform (PTP) is intended to address a major deficiency in the development of parallel programs: the lack of a robust open-source tegrated development environment designed to provide a range of tools that can assist in software development for parallel applications. PTP offers a variety of useful features for parallel application developers, in particular: parallel IDE, scalable debugger, integration with parallel tools or interaction with parallel systems. Currently, main supported languages are C/C++ and Fortran. Thus, we used the Eclipse PTP framework to support QCG-OMPI and remote job submission, debugging and monitoring using the QosCosGrid middleware, in particular QCG-Computing services providing OGSA-BES interface.

Attachments