The ascendance of networked information in our economy and daily lives has increased the awareness of the importance of dependability features. OLTP (On-Line Transaction Processing) systems constitute the kernel of the information systems used today to support the daily operations of most of the business. Although these systems comprise the best examples of complex business-critical systems, no practical way has been proposed so far to characterize the impact of faults in such systems or to compare alternative solutions concerning dependability features. This paper proposes a dependability benchmark for OLTP systems. This dependability benchmark uses the workload of the TPC-C performance benchmark and specifies the measures and all the steps required to evaluate both the performance and key dependability features of OLTP systems, with emphasis on availability. This dependability benchmark is presented through a concrete example of benchmarking the performance and dependability of several different transactional systems configurations. The effort required to run the dependability benchmark is also discussed in detail.
Subject
Dependability benchmarking
Related Project
DBench - Dependability Benchmarking
Conference
Very Large Databases Conference - VLDB2003, September 2003
PDF File
Cited by
Year 2013 : 5 citations
Domenico Cotroneo, Roberto Natella, Stefano Russo, Fabio Scippacercola, "State-Driven Testing of Distributed Systems", 17th International Conference on Principles of Distributed Systems, OPODIS 2013, Nice, France, December 16-18, 2013.
Alessandro Gustavo Fior, Jorge Augusto Meira, Eduardo Cunha de Almeida, Ricardo Gonçalves Coelho, Marcos Didonet Del Fabro, Yves Le Traon, “Under Pressure Benchmark for DBMS Availability”, Journal of Information and Data Management, Vol. 4 No. 3, 2013.
Alessandro Gustavo Farias Fior, "Under Pressure Benchmark: A Large-Scale Availability Benchmark For Distributed Databases", Msc Thesis, Universidade Federal Do Paraná, Curitiba, Brazil, 2013.
Roberto Natella, Fabio Scippacercola, "Issues and Ongoing Work on State-Driven Workload Generation for Distributed Systems", 14th European Workshop on Dependable Computing, EWDC 2013, Coimbra, Portugal, May 14-15, 2013.
Domenico Di Leo, “Robustness Evaluation of Software Systems through Fault Injection”, PhD Thesis, Universit a degli Studi di Napoli Federico II, Italy, 2013.
Year 2012 : 5 citations
Fabio Scippacercola, "State-Driven Workload Generation in Distributed Systems, Tesi di Laurea Magistrale, Universita' Degli Studi di Napoli Federico II, Naples, Italy, 2012.
Amit Sangroya, Damián Serrano, Sara Bouchenak, "Benchmarking Dependability of MapReduce Systems", 31st IEEE International Symposium on Reliable Distributed Systems, Irvine, CA, USA, October 2012.
Seung-Ho Lim, Hyun Jin Choi, Doo-Soon Park, "Efficient journaling writeback schemes for reliable and high-performance storage systems", Personal and Ubiquitous Computing, September 2012.
Domenico Cotroneo, Francesco Fucci, Roberto Natella, "Towards a State driven Workload Generation Framework for Dependability Assessment", The Fifth International Conference on Dependability, DEPEND 2012, Rome, Italy, August 19 - 24, 2012.
Anja Bog, "Benchmarking Composite Transaction and Analytical Processing Systems", University of Potsdam/Hasso-Plattner-Institut, Potsdam, Germany, June 2012.
Year 2011 : 4 citations
Roberto Natella, “Achieving Representative Faultloads in Software Fault Injection”, PhD Thesis, Universita' Degli Studi di Napoli Federico II, Italy, November 2011.
Yantao Li, Charles Levine, “Extending TPC-E to Measure Availability in Database Systems”, Third TPC Technology Conference on Performance Evaluation and Benchmarking, TPCTC 2011, Seattle, WA, USA, August 29, 2011.
Bo Li, Haiying Zhou, Decheng Zuo, Zhan Zhang, Peng Zhou, Long Jia, "Performance modeling and benchmarking of bank intermediary business on high-performance fault-tolerant computers", 2011 IEEE/IFIP 41st International Conference on Dependable Systems and Networks Workshops, Hong Kong, June 2011.
A. Bondavalli, P. Lollini, A. Bovenzi, M. Colajanni, L. Coppolino, C. Esposito, M. Fixxo, C. di Martino, L. Montecchi, R. Natella, A. Peccia, "D3.1: Modeling and Evaluation: State-of-the-art", deliverable of the project: Dependable Off-The-Shelf based Middleware Systems for Large-scale Complex Critical Infrastructures (DOTS-LCCI Project: PRIN 2008LWRBHF), Italy, March 2011.
Year 2010 : 6 citations
4. Fábio Oliveira, "Towards Mistake-Aware Systems", PhD Thesis, Graduate School—New Brunswick Rutgers, The State University of New Jersey, October 2010.
5. Philipp Reinecke, Katinka Wolter, Miroslaw Malek, “#A Survey on Fault-Models for QoS Studies of Service-Oriented Systems”, Technical Report number B-2010-02, Freie Universität Berlin, February, 2010.
6. Manuel José Ferreira Carneiro Mendonça, "Vulnerability Detection in Device Drivers", PhD Thesis Proposal, University of Lisbon, Portugal, January 2010.
1. Anja Bog, Hasso Plattner, Alexander Zeier, "A mixed transaction processing and operational reporting benchmark", Information Systems Frontiers, Springer Netherlands, ISSN: 1387-3326, November 2010.
2. Goetz Graefe, Arnd Christian Konig, Harumi Kuno, Volker Markl, Kai-Uwe Sattler, "Robust Query Processing", Abstracts Collection and Executive Summary - Dagstuhl Seminar 20381, Dagstuhl, September 2010.
3. Bo Li, Haiying Zhou, Decheng Zuo, Zhan Zhang, "Study for Performance Benchmark of Bank Intermediary Business on High-Performance Fault-Tolerant Computers”, International Symposium on Parallel and Distributed Processing with Applications, ISPA 2010, Taipei, Taiwan, September 2010.
Year 2009 : 4 citations
1. Hongfei Guo Dan Jones Jennifer Beckmann Praveen Seshadri, “Declarative Management in Microsoft SQL Server”, 35th International Conference on Very Large Databases, VLDB09, Lyon, France, August 24-28, 2009.
2. Tsz-kin Lau, Peter Kin Leung Shum, "Quantitative measurement of the autonomic capabilities of computing systems", International Business Machines Corporation (IBM), United States Patent 7539904, May 2009.
3. Goetz Graefe, Harumi A. Kuno, Janet L. Wiener, “Visualizing the robustness of query execution”, Fourth Biennial Conference on Innovative Data Systems Research, CIDR 2009, Asilomar, CA, USA, January 4-7, 2009.
4. Choi, H. J., Lim, S., and Park, K. H., “JFTL: A flash translation layer based on a journal remapping for flash memory”, ACM Transactions on Storage, Vol 4, Issue, January 2009.
Year 2008 : 4 citations
1. Gustavo M. D. Vieira, Willy Zwaenepoel, Luiz E. Buzato, “Dynamic Content Web Applications: Crash, Failover, and Recovery Analysis”, Technical Report - IC-08-34, Instituto de Computacão, University of Campinas, Campinas, Brazil, December 2008.
2. A. Bog, J. Krueger, J. Schaffner, “A Composite Benchmark for Online Transaction Processing and Operational Reporting”, IEEE Symposium on Advanced Management of Information for Globalized Enterprises (AMIGE'08), Tianjin, China, 2008.
3. Imre Kocsis, Zsolt Kocsis, Imre Malatinszki, András Pararicza, “Dependabilty Benchmarks and System Supervision”, SRDS 2008 Workshop on Sharing Field Data and Experiment Measurements on Resilience of Distributed Computing Systems (jointly organized with the 27th International Symposium on Reliable Distributed Systems, IEEE SRDS 2008), Naples, Italy, October 5, 2008.
4. Sung Hoon Baek, Kyu Ho Park, "Prefetching with Adaptive Cache Culling for Striped Disk Arrays", The 2008 USENIX Annual Technical Conference, Boston, USA, June 25-27, 2008.
Year 2007 : 6 citations
1. Gergely Pintér, “Model Based Program Synthesis and Runtime Error Detection for Dependable Embedded Systems”, PhD Thesis, Department of Measurement and Information Systems, Budapest University of Technology and Economics, Budapest, Hungary, 2007.
2. M. Banatre, A. Pataricza, A. van Moorsel, P. Palanque, L. Strigini, “From Resilience-Building to Resilience-Scaling Technologies: Directions – ReSIST NoE Deliverable D13”, deliverable of the project: ReSIST: Resilience for Survivability in IST - A European Network of Excellence (ReSIST: Contract Number: 026764), September 2007.
3. S. H. Baek, K. H. Park, “Matrix-Stripe-Cache-Based Contiguity Transform for Fragmented Writes in RAID-5”, IEEE Transactions on Computers, Volume 56 , Issue 8 , August, 2007.
4. Manuel Mendonça, Nuno Neves, “Robustness Testing of the Windows DDK”, IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2007, Edinburgh, UK, June 25-28, 2007.
5. David C.C. Ong, Rytis Sileika, Souheil Khaddaj, Radouane Oudrhiri, “Alternative data storage solution for mobile messaging services”, Mobile Information Systems, Volume 3, Number 1, pp. 39 – 54, 2007.
6. André Cardeal Santana; Sueli de Fatima Poppi Borba, “Desempenho de Dados em Modelos de Relacionamentos por Herança”, Campo Digit@l, ISSN 1981-092X, vol.2 nº1, 2007.
Year 2006 : 2 citations
1. Shih-Han Chiu, "Toward a More Generalized Benchmark Workload Model for Data Warehouse and Data Mining", PhD Thesis, National Chengchi University, 2006.
2. Farès Saad Khorchef, “Un Cadre Formel pour le Test de Robustesse des Protocoles de Communication”, PhD Thesis, LAAS-CNRS, Toulouse, France, December 2006.
Year 2005 : 2 citations
1. Ali Kalakech, “Étalonnage de la sûreté de fonctionnement des systèmes d’exploitation – Spécifications et mise en œuvre”, PhD Thesis, LAAS-CNRS, Toulouse, France, August 2005.
2. F. Oliveira, K. Nagaraja, R. Bachwani, R. Bianchini, R. P. Martin, and T. Nguyen. "Understanding and Validating Database System Administration", 2005 USENIX Annual Technical Conference, USENIX '05, May 30-June 3, 2005.
Year 2004 : 1 citations
1. Ali Kalakech, Karama Kanoun, Yves Crouzet, Jean Arlat, “Benchmarking The Dependability of Windows NT4, 2000 and XP”, IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2004, Florence, Italy, June 28-July 1, 2004.
