A Framework for Performance Evaluation of Complex Event Processing Systems (poster)
Authors
Abstract
Several new Complex Event Processing (CEP) engines have been recently released, many of which are intended to be used in performance sensitive scenarios - like fraud detection, traffic control, or health care systems. However, there is no standard means to assess the performance of a CEP engine. This omission is all the more relevant as there are currently many competing products, languages, architectures, data models, and data processing CEP techniques. A performance evaluation framework can help identify good design decisions and assist in improving engines. Here we demonstrate our work in progress: FINCoS, a framework that can be used to benchmark CEP systems. The proposed framework has five relevant characteristics:i. Flexible (e.g., it allows changing the workload on the fly to measure reactions to peak loads);
ii. Independent of particular workloads;
iii. Neutral (not bound to any specific CEP product);
iv. Correctness check (validators can be plugged into the framework on demand to verify results);
v. Scalable (many of its components, like event generators, can be centrally orchestrated and run in parallel).
Note that the framework does not include a benchmark specification. In fact, it was designed such that diverse datasets and query scenarios can be easily attached and tested on several CEP engines.
As such, this framework has three key benefits: first, it can be used by the CEP community to more quickly devise and experiment new benchmarks for event processing systems. Second, CEP vendors can employ the framework in conjunction with their own test datasets to benchmark their systems internally. Finally, customers can use it with their real data and select the CEP engine that best fits their needs.