HPCoLSI Project - High Performance Computing over the Large-Scale Internet

Description

This project advances the current state of the art in platforms for Internet-wide computation, by designing, implementing and evaluating new mechanisms that move beyond the traditional client-server architecture of these platforms to support, for the first time, scalable decentralized cooperation among clients.   Recent research and development in massive desktop grids has enabled the creation of a computing infrastructure based on voluntary donation of compute cycles that has the potential to become the most powerful computing platform on Earth: public volunteer computing, using the BOINC middleware, can deliver over 1000 TFLOPS as of 2008.   Today, our efforts to take full advantage of this impressive computational power are fundamentally hampered by one limitation: all computing infrastructures based on volunteer computing only supports client-server architectures. In this approach, the required computation is subdivided by the server in a set of mutually independent tasks that are then farmed out to workers (the clients) who, upon completing their assignment, report the result back to the server.   While a client-server architecture is very effective for embarrassingly parallel applications such as SETI@home, it is structurally unable to support the broad range of applications that can take advantage of an internet-wide volunteer-based computing infrastructure, as there is no inter-client communication.   This project addresses the key challenge of transitioning from the centralized architecture to one that is distributed and scalable. By adding communication between clients, we may remove the central server from the communication loop, for the sake of economy and speed. Whenever possible, direct (or delayed) exchange of data among clients may save precious bandwidth resources from the project ?s owner. Additionally, for applications with controlled IO requirements, inter-client communication may speedup execution. To support this goal, most of our work will focus on two aspects: one is to prepare the BOINC middleware for the change, the other is to build on P2P computation to achieve some guarantees in such a hostile environment.   Extending BOINC will require to have better data on the clients, new scheduling algorithms for parallel tasks, new algorithms to ensure trustworthiness of computation, and the adoption of parallel programming paradigms. On the other hand, adopting a P2P approach will enable to make groups of nodes that are alike, to ensure timely communication among them (as we can find in successful P2P streaming systems), run decentralized groups across different administrative domains, store and retrieve data from the community and on top of that to achieve dependability of these services even under malicious, broken, or otherwise byzantine action.   To illustrate the benefits of the solutions developed during the project, we will build a parallel evolutionary algorithm, to solve a real-world chemistry problem.

Researchers

Funded by

FCT

Partners

IST

Total budget

127 699,00 €

Keywords

Desktop Grid, BOINC

Start Date

2010-05-01

End Date

2013-04-30

Conference Articles