Download Chapter 1 Introduction to EverBEEN

CHAPTER 1. INTRODUCTION TO EVERBEEN
6
• The environment can offer facilities, such as synchronization, logging and communication mechanisms,
that would make the task of creating a benchmark easier.
• Statistics, analysis and visualization are good candidates for having a helpful library instead of
writing a custom one.
1.1.1
Related works
• Kalibera, T., Lehotsky J., Majda D., Repcek B., Tomcanyi M., Tomecek A., Tuma, P., Urban J.:
Automated Benchmarking and Analysis Tool1 (PDF, 149 kB), VALUETOOLS 2006
• Kalibera, T., Bulej, L., Tuma, P.: Generic Environment for Full Automation of Benchmarking2
(PDF, 84 kB), SOQUA 2004
• Täuber, J.: Deployment of Performance Evaluation Tools in Industrial Use Case3 (PDF, 614 kB),
MFF 2013
1.2
Case study
During the development of EverBEEN, several use cases were considered and this section describes the
ones that BEEN was specifically designed for.
1.2.1
Regression benchmarking
Regression benchmarking is a technique mostly aimed at discovering negative performance impact of a
newly added feature, an upgrade or a single patch in the source code. This usually involves performing
the same set of benchmarks against various versions of the same software. Often, even individual revisions
of the source tree are tested. When the benchmark is stable enough to result in consistent data, it is easy
to immediately see which commits had an impact on performance.
While performance degradation is obviously an undesired effect, unplanned performance increase can
also be an indicative of a problem. This can easily happen when an expensive check (e.g. for security
purposes) is unintentionally removed or bypassed. Whether performance fluctuations be positive or
negative, regression benchmarking provides the development team with crucial information which not
only denotes the actutal performance change, but also points to the exact code modification that caused
it.
1.2.2
Pull-oriented benchmarking
Consider the following case: We have access to a source code repository with version history of a software
product, whose developers don’t do any regular benchmarking. Suddenly, they realize that their software
behaves much slower than a year ago. Although the performance degradation is probably caused by
several factors, the developers would like to determine major slowdown culprits and eliminate them.
If this software were a standalone desktop application, the obvious solution would be to build all the
revisions of the software since last year and benchmark them. Writing a script automating this task
would be straightforward. When the software in question is distributed middleware, even setting up a
benchmarking environment could be a costly task.
This task can be generalized into a problem of running a benchmark over a set of parameters. The
parameters are known in advance and the benchmark is by necessity a user-written code. A generic
benchmarking framework should therefore simplify both parameter specification and the process of writing
benchmark code. The user’s options must be flexible enough to support many possible configurations of
the benchmark — one might want to benchmark a single piece of software with various configurations.
1 http://d3s.mff.cuni.cz/publications/download/Submitted_1404_BEEN.pdf
2 http://d3s.mff.cuni.cz/publications/download/KaliberaBulejTuma-FullAutomationOfBenchmarking.pdf
3 https://is.cuni.cz/webapps/zzp/detail/78663/4417375