CISUC

Optimizing distributed file storages and processing engines for CERN’s Large Hadron Collider using multi criteria partitioned replication

Authors

Abstract

Throughout the last decades, distributed file systems and process-
ing engines have been the primary choice for applications requir-
ing access to large amounts of data. Since the introduction of the
MapReduce paradigm, relational databases are being increasingly
replaced by more efficient and scalable architectures, in particular
in environments where a query is expected to process TBytes or
even PBytes of data in a single execution. That is the situation at
CERN, where data storage systems that are critical for the safe
operation, exploitation and optimization of the particle accelerator
complex, are based on traditional databases or file system solutions,
which are already working well beyond their initially provisioned
capacity. Despite the efficiency of modern distributed data storage
and processing engines in handling large amounts of data, they are
not optimized for heterogeneous workloads such as they arise in
the dynamic environment of one of the world’s largest scientific
facilities.
This contribution presents a Mixed Partitioning Scheme Replica-
tion (MPSR) solution that outperforms the conventional distributed
processing environment configurations at CERN for virtually the
entire parameter space of the accelerator monitoring systems’ work-
load variations. Our main strategy was to replicate the data using
different partitioning schemes for each replica, whereas the individ-
ual partitioning criteria is dynamically derived from the observed
workload. To assess the efficiency of this approach in a wide range
of scenarios, a behavioral simulator has been developed to com-
pare and analyze the performance of the MPSR with the current
solution. Furthermore we present the first actual results of the
Hadoop-based prototype running on a relatively small cluster that
not only validates the simulation predictions but also confirms the
higher efficiency of the proposed technique.

Keywords

Mixed Partitioning Scheme Replication, MPSR, Distributed databases, Large-Hadron collider, CERN

Subject

Distributed File Storage

Conference

International Conference on Distributed Computing and Networking, ICDCN 2018, January 2018


Cited by

No citations found