InfoSizing

The TPC provides a specification document that contains a functional description of what should be implemented. The TPC does not provide a full implementation kit for its benchmarks. Each vendor is responsible for designing and building an optimal implementation of the TPC's functional specifications. (In the case of TPC-E and TPC-H some code is provided by the TPC to control the database population and the user input.)

At the center of any TPC benchmark implementation is a DBMS. A large portion of the implementation is dependent on the DBMS. As a result, most DBMS vendors have developed their own benchmark implementation kit. These kits vary in completeness and ease of use. Some are better adapted to selected operating systems.

The first step toward implementing a TPC benchmark is to select a DBMS vendor. Aside from strategic relationships, one of the criteria for selecting that vendor should be how complete, adapted and easy to use their TPC implementation kit is and how willing they are to support the benchmarking effort.

Once a DBMS has been selected and an implementation kit obtained from the DBMS vendor, the implementation can be deployed on the target hardware platform.

top of page

The first deployment of any TPC benchmark kit on a target hardware platform will typically produce mediocre performance results. To bring the performance results to a competitive level, much tuning is required.

There is no guideline on how much tuning is enough. It all depends on many factors involving the hardware and the software components of the test platform as well as the performance goals that have been established.

At one end of the spectrum, the target hardware platform has no significant flaw and is well adapted to the TPC benchmark kit and the underlying DBMS. A few weeks of tuning usually produces the desired results.

On the other end of the spectrum, the first few executions of the benchmark may uncover major flaws in the hardware design. Some components such as the I/O bus or the CPU caching architecture may have to be re-engineered. The port of the DBMS on the hardware platform may be deficient and may have to be re-optimized. Several months of work may be necessary before an acceptable performance result can be produced.

top of page

The TPC requires that all published results be independently verified by a TPC-Certified Auditor. The first step of this verification is the audit of the benchmark implementation.

This step is also referred to as the "pre-audit" as it takes place prior to the audit of the actual performance result. During the pre-audit, all the components of the implementation are verified for conformance with the TPC benchmark specification.

In addition, all the procedures and scripts that will be used to execute the actual performance measurements are reviewed and approved by the auditor. The benchmark is now ready to be executed and compliant performance results can be produced.

top of page

Once all major bugs and performance deficiencies have been removed and once the benchmark implementation has been audited, the final tuning and the formal measurements can take place.

The final tuning usually consists of finalizing the layout of the database on the available disks and fine-tuning the various DBMS and OS parameters. Last minute bugs may also be discovered and may have to be corrected at this stage.

Because the implementation has already been audited, no major changes can be made at this point without having to repeat the pre-audit.

Once the target performance has been reached, a fresh new database in built and the formal measurements begin. The audited procedures are followed and the audited scripts are executed. The results of all the measurement tests are captured into predefined audit files, ready for the final audit.

top of page

The results from all the measurement tests must be audited by a TPC-Certified Auditor. This is usually done by sending the entire set of final audit files to the auditor. This process is referred to as a "remote audit" as it does not require the physical presence of an auditor during the actual test.

The auditor verifies the results from the measurement tests by examining the audit files. In addition, the auditor verifies the configuration used in the test and checks that the pricing of the 3-year cost of ownership complies with the benchmark requirements. When necessary, the auditor has the option of remotely connecting to the test bed for additional hands-on verifications.

top of page

Only fully audited results can be disclosed to the TPC. The TPC's disclosure requirements include the production and submittal of an FDR.

An FDR must be produced for each benchmark result. This report must be sent to the TPC prior to any public use of the result. It must include the details of the entire implementation, price quotes for third party components used in the configuration and an attestation letter from a TPC-Certified Auditor.

Each FDR, once filed with the TPC, enters a 60-day review period during which it can be challenged for non-compliance with the benchmark specifications. While a few FDRs are challenged from time to time, it should be mentioned that since mandatory audits have been established even fewer results have been declared non-compliant.

top of page