Experimental Dependability Evaluation

Authors

Abstract

Computer systems dependability has to be easy to measure to become a commodity. This is not yet the case, as there are still significant uncertainties at several levels of the evaluation process. In this chapter we start by describing the available experimental techniques that help in that process, like fault injection, robustness testing and field measurement. The capabilities and limitations of each of these techniques is presented, and the main dimensions required for them to participate in systematic dependability benchmarking are discussed, like how to build representative evaluation setups, choosing workloads and faultloads that are representative, portable and repeatable, and choosing the right measures to obtain meaningful results. After this, still only characterization data on fault tolerance methods will be available; to obtain global dependability measures like availability and reliability a modelling phase is required, where adequate fault rates are considered. We discuss also the development phase of a system where each of the previous techniques is applied, and end by presenting our opinion on the most important research directions for experimental dependability benchmarking.

Keywords

experimental dependability evaluation, fault injection, dependability benchmarking, field measurement, fault emulation, operator faults, software faults, hardware faults, robustness testing

Subject

Dependability Evaluation and Analysis

Book Chapter

Dependable Computing Systems: Paradigms, Performance Issues, and Applications, 12, John Wiley & Sons, Inc, November 2005