InfoSizing

Custom benchmarks are at one end of the design choice spectrum. Custom benchmarks will usually best represent target application performance. However, this accuracy comes with a cost. The definition of a fully custom benchmark often turns out to be too time consuming and costly. In addition, there is a high risk of producing a biased test that inadvertently favors one vendor over the others.

Semi-Standard benchmarks represent the middle ground in the design choice spectrum. Semi-Standard benchmarks attempt to capture the best of both custom and standard designs while avoiding thier associated pitfalls.

Standard benchmarks are at the other end of the design choice spectrum. Starndard benchamrks completely eliminate the time and cost of custom designs. In addition, it often provides the highest degree of impartiality toward all vendors. However, the use of a standard benchmark also means that the specific needs of the target application are not examined and therefore the winning technology may not actually be the most suitable.

With a semi-standard benchmark, most of the benchmark design and definition is eliminated by the adoption of an existing standard benchmark. The work is focused on an analysis on the specificity of the target application, which gets translated into a set of elements of the standard benchmark that need to be adapted, augmented or eliminated.

Adapting an existing benchmark requires substantially less time and effort than the complete definition of a fully custom benchmark. Yet, experience has shown that the quality of the outcome is generally equal if not superior.

The option of a semi-standard benchmark was chosen by such data warehouse pioneers as Boeing and MCI as the premier tool in their technology selection process. By using an adapted version of the TPC-H standard they were successful in combining the specificity of a custom approach with the reliability of a standard.