Changeloads: a Fundamental Piece on the SASO Systems Benchmarking Puzzle

Authors

Abstract

Benchmarks have been traditionally tailored to static, unchangeable systems, functioning in well-known and controlled environments. Thus, established benchmarks (and benchmarking approaches) are becoming progressively less representative of real world scenarios, as change is gaining emphasis as a fundamental player in computing systems runtime conditions. As today’s systems are becoming, at the very least, reactive to changes (either endogenous or exogenous) at some level, if not even proactive in reaching their goals more efficiently and effectively, we believe that benchmarks must also evolve, becoming applicable to systems that react to change, adapt, evolve, and have the capability to improve their own performance. In this position paper, we argue that representative changeloads are now as vital as representative workloads, and that changeload-based benchmarks will become a key part in the development and evaluation of SASO systems. We present and discuss some applications of benchmarks in this area, proposing some directions for research.

Keywords

benchmarking, changeload, resilience, performance, self-adaptive, self-organizing, autonomic systems

Subject

Benchmarking of Self-Adaptive and Self-organizing Systems

Related Project

ADAAS: Assuring Dependability in Architecture-based Adaptive Systems

Conference

International Workshop on Evaluation for Self-Adaptive and Self-Organizing Systems (Eval4SASO) 2012

PDF File