The following text
is updated regulary by active SPEC members. Last update: 09/11/1995 (Changes to previous version: SPEC member list, Order Form)
Legally, SPEC is a non-profit corporation registered in California.
SPEC basically performs two functions:
Standard Performance Evaluation Corporation (SPEC) 2722 Merrilee Drive, Suite 200 Fairfax, VA 22031 USA Phone: +1-703-698-9604 FAX: +1-703-560-2752 E-Mail: email@example.comDianne Rice (ext. 325) is the Director of Operations for SPEC and Rodney Sanford (ext. 318) is the SPEC administrator. They are responsible for the adminstration of SPEC's products and for assisting customers with questions and orders. Technical questions regarding the SPEC benchmarks (e.g., problems with execution of the benchmarks), are usually referred to SPEC's technical support person..
The SPEC benchmark sources are generally available, but not free. SPEC is charging separately for its benchmark suites. The income from the benchmark source tapes is intended to support the administrative costs of the corporation. benchmarks, and so on. Buyers of the benchmark tapes receive a shrink-wrapped site license with their first order of any of SPEC's tape products along with the manual explaining the rules for result publications. All benchmark suites come on QIC 24 tapes, written in UNIX tar format.
Current prices are:
CINT92 $ 425.00 (CPU intensive integer benchmarks) CFP92 $ 575.00 (CPU intensive floating point benchmarks) CINT92&CFP92 $ 900.00 SDM $ 1450.00 (UNIX Software Development Workloads) SFS $ 1200.00 (System level file server (NFS) workload)* Accredited universities receive a 50 percent discount on SPEC tape products.
The SPEC Newsletter is published quarterly and contains result publications for a variety of machines, about 50-70 pages of result pages per issue, with articles dealing with SPEC and benchmarking.
SPEC Newsletter Prices (1 year subscription, 4 issues)
$ 550.00 for USA Orders $ 575.00 for International Orders
There are currently two suites of compute-intensive SPEC benchmarks, measuring the performance of CPU, memory system, and compiler code generation. They normally use UNIX as the portability vehicle, but they have been ported to other operating systems as well. The percentage of time spent in operating system and I/O functions is generally negligible.
This suite contains six benchmarks performing integer computations, all of them are written in C. The individual programs are:
Number and name Application Approx. size gross net 008.espresso Logic Design 14800 11000 022.li Interpreter 7700 5000 023.eqntott Logic Design 3600 2600 026.compress Data Compression 1500 1000 072.sc Spreadsheet 8500 7100 085.gcc Compiler 87800 58800 ------ ----- 123900 85500The approximate static size is given in numbers of source code lines, including declarations (header files). "Gross" numbers include comments and blank lines, "net" numbers exclude them.
A somewhat more detailed, though still short description of the benchmarks (from an article by Jeff Reilly, in SPEC Newsletter vol. 4, no. 4):
008.espresso Generates and optimizes Programmable Logic Arrays. 022.li Uses a LISP interpreter to solve the nine queens problem, using a recursive backtracking algorithm. 023.eqntott Translates a logical representation of a Boolean equation to a truth table. 026.compress Reduces the size of input files by using Lempel-Ziv coding. 072.sc Calculates budgets, SPEC metrics and amortization schedules in a spreadsheet based on the UNIX cursor- controlled package "curses". 085.gcc Translates preprocessed C source files into optimized Sun-3 assembly language output.
Number and name Application Lang. Approx. size gross net 013.spice2g6 Circuit Design F 18900 15000 015.doduc Simulation F 5300 5300 034.mdljdp2 Quantum Chemistry F 4500 3600 039.wave5 Electromagnetism F 7600 6400 047.tomcatv Geometric Translation F 200 100 048 ora Optics F 500 300 052.alvinn Robotics C 300 200 056.ear Medical Simulation C 5200 3300 077.mdljsp2 Quantum Chemistry F 3900 3100 078.swm256 Simulation F 500 300 089.su2cor Quantum Physics F 2500 1700 090.hydro2d Astrophysics F 4500 1700 093.nasa7 NASA Kernels F 1300 800 094.fpppp Quantum Chemistry F 2700 2100 ----- ----- 57900 43900Short description of the benchmarks:
013.spice2g6 Simulates analog circuits (double precision). 015.doduc Performs Monte-Carlo simulation of the time evolution of a thermo-hydraulic model for a nuclear reactor's component (double precision). 034.mdljdp2 Solves motion equations for a model of 500 atoms interacting through the idealized Lennard-Jones potential (double precision). 039.wave5 Solves particle and Maxwell's equations on a Cartesian mesh (single precision). 047.tomcatv Generates two-dimensional, boundary-fitted coordinate systems around general geometric domains (vectorizable, double precision). 048 ora Traces rays through an optical surface containing spherical and planar surfaces (double precision). 052.alvinn Trains a neural network using back propagation (single precision). 056.ear Simulates the human ear by converting a sound file to a cochleogram using Fast Fourier Transforms and other math library functions (single precision). 077.mdljsp2 Similar to 034.mdljdp2, solves motion equations for a model of 500 atoms (single precision). 078.swm256 Solves the system of shallow water equations using finite difference approximations (single precision). 089.su2cor Calculates masses of elementary particles in the framework of the Quark Gluon theory (vectorizable, double precision). 090.hydro2d Uses hydrodynamical Navier Stokes equations to calculate galactical jets (vectorizable, double precision). 093.nasa7 Executes seven program kernels of operations used frequently in NASA applications, such as Fourier transforms and matrix manipulations (double precision). 094.fpppp Calculates multi-electron integral derivatives (double precision).More information about the individual benchmarks is contained in description files in each benchmark's subdirectory on the SPEC benchmark tape.
The CPU benchmarks can be used for measurement in two ways:
As is apparent from results publications, the different SPEC ratios for a given machine can vary widely. SPEC encourages the public to look at the individual results for each benchmarks. Users should compare the characteristics of their workload with that of the individual SPEC benchmarks and consider those benchmarks that best approximate their jobs. However, SPEC also recognizes the demand for aggregate result numbers and has defined the integer and floating-point averages
SPECint92 = geometric mean of the six SPEC ratios from CINT92 SPECfp92 = geometric mean of the 14 SPEC ratios from CFP92For "baseline" measurements (see below), the averages are
SPECbase_int92 = geometric mean of the six SPEC ratios from CINT92, under the baseline measurement rules SPECbase_fp92 = geometric mean of the 14 SPEC ratios from CFP92, under the baseline measurement rules
Similar as with the speed metric, SPEC has defined averages
SPECrate_int92 = geometric mean of the 6 SPEC rates from CINT92 SPECrate_fp92 = geometric mean of the 14 SPEC rates from CFP92For "baseline" measurements (see below), the averages are
SPECrate_base_int92 = geometric mean of the 6 SPEC rates from CINT92, under the baseline measurement rules SPECrate_base_fp92 = geometric mean of the 14 SPEC rates from CFP92, under the baseline measurement rulesBecause of the different units, the values SPECint92/SPECfp92 and SPECrate_int92/SPECrate_fp92 cannot be compared directly.
The detailed baseline rules, together with an explanation, are listed in an article "Reviewing the New Baseline Rules" (R. Weicker, J. Reilly) in SPEC Newsletter vol. 6 no. 2 (June 1994).
The intention is that these results represent the performance a not so sophisticated user would achieve, whereas the traditional rules allow a selection of optimization flags that is more typical for sophisticated users. As a general guideline, a system vendor is expected to endorse the general use of these baseline options by customers who seek to achieve good application performance.
Effective June 1994, when SPEC's CPU benchmark results are reported, the reports have to include baseline results. Baseline-only reporting is allowed, but since the non-baseline level is equivalent to the previous reporting level, it can be expected that most companies will report numbers for both optimization levels.
057.sdet Represents a large commercial UNIX/C based software development environment. This characterization is based on AT&T analysis and models developed by Steve Gaede, formerly with AT&T Bell Laboratories. 061.kenbus1 Represents UNIX/C usage in a Research and Development environment. This characterization is based on data collection and analysis at Monash University by Ken McDonell.For each benchmark, throughput numbers (scripts, i.e. simulated user loads per hour) are given for several values of concurrent workloads. The reader can determine the peak throughput as well as the ability of a system to sustain throughput over a range of concurrent workloads. Since the workloads for the two benchmarks are different, their throughput values are also different and cannot be compared directly.
SFS Release 1.1 contains one benchmark, 097.LADDIS. This benchmark measures NFS file server performance in terms of NFS response time and throughput. It does this by generating a synthetic NFS workload based on a workload abstraction of an NFS operation mix and an NFS operation request rate.
Running 097.LADDIS requires a file server (the entity being measured) and two or more "load generators" connected to the file server via a network medium. The load generators are each loaded with 097.LADDIS and perform the 097.LADDIS workload on file systems exported by the file server.
SFS Release 1.0 results include full server configuration information (hardware and software) and a graph of server response time versus NFS load for the 097.LADDIS operation mix. Compared with the predecessor version SFS 1.0, Release 1.1 contains bug fixes and improved documentation. The workload did not change, so results are comparable.
SPECint89 = geometric mean of the SPEC ratios of the 4 integer programs in rel. 1.2b (CPU-Suite of 1989). SPECfp89 = geometric mean of the SPEC ratios of the six floating-point programs in rel. 1.2b. SPECmark89 = geometric mean of all 10 SPEC ratios of the programs in rel. 1.2b.In addition, there was the possibility of throughput measurements, with two copies of a benchmark running per CPU, called "Thruput Method A" (There was never a "Method B"). The following average values had been defined:
SPECintThruput89 = geometric mean of the Thruput Ratios of the four integer programs SPECfpThruput89 = geometric mean of the Thruput Ratios of the six floating-point programs SPECThruput89 ("aggregate thruput") = geometric mean of the Thruput Ratios of all 10 programsSPEC now discourages use of the 1989 benchmark suite and recommends use of the CINT92 and CFP92 suites, for the following reasons:
SPEC discontinued selling SPEC Release 1.2b tapes after December 1992; labeled any the 1.2b results published in the SPEC Newsletter with "Benchmark Obsolete". SPEC discontinued result publications for this suite in June 1993. A similar obsoletion process can be expected for the 1992 CPU benchmarks once the 1995 CPU benchmarks are in place (see below; section 6.2).
SPEC (SFSSC) is also working on a new version of the SFS benchmark suite.
In May 1995, a final vote of all SPEC OSG member companies finalized the benchmark selection process, all 8 integer and 10 floating-point benchmarks proposed by the Open Systems Steering Committee were accepted. First results obtained with the new suites are expected to accompany the announcement (probably August 1995), first newsletter results are anticipated for publication in the September 1995 SPEC Newsletter.
The run and reporting rules for the new benchmarks suites are currently being finalized. It can be expected that again there will be a peak and a baseline metric. However, details will be different from the current rules.
The new SPEC CPU benchmarks will also be ported to the Windows
NT operating system. Since very few of the CPU benchmarks make
operating system calls (they intentionally measure CPU / Memory /
Compiler performance only), the two versions cannot be used to compare
the efficiency of different operating systems. This port will, however,
add a new portability vehicle to the UNIX systems that are used most
The group has elected its own officers and will develop its own benchmarks, together will appropriate run and reporting rules. Since the effort has just started in 1994, no selection of benchmarks has been made yet. Several benchmark candidate codes are being evaluated by the new group.
To distinguish the two activities that now exist under the SPEC umbrella, the "traditional" SPEC group has now been renamed the Open Systems Group (with the Open Systems Steering Committee as the body for technical decisions), the new group has the name High Performance Computing Group.
Annual Dues $ 5000.00 Initiation Fee $ 1000.00The dues for membership in OSG and HPG are separate.
There is also the category of a "SPEC Associate", intended for accredited educational institutions or non-profit organizations:
Annual Dues $ 1000.00 Initiation Fee $ 500.00Among the membership benefits are Newsletters and benchmark tapes as they are available, with company-wide licenses within their countries. Most important is early access to benchmarks that are being developed, and the possibility to participate in technical work on the benchmarks.
The benefits for associates are the same as for members except that OSG associates have no voint rights. The intention for associates is that they can act in an advisory capacity to SPEC, getting first-hand experience in an area that is widely neglected in academia but nonetheless very important in the "real world", and providing technical input to SPEC's task.
SPEC meetings (Open Systems Group and SFS subgroup) are held about every two months, for technical work and decisions about the benchmarks. The SPEC High Performance Computing group meets four times in the year.
Every member or associate can participate in these meetings and make proposals. In the Open Systems Group, decisions are made by the Open Systems Steering Committee (nine members) elected by the general membership at the Annual Meeting. In the High Performance Computing Group, the Steering Committee consists of all members present at a meeting. All SPEC members of OSG or HPG vote before a benchmark is finally accepted.
|Item No.||Products||Unit Cost|| Educational**
(To order back issues, contact the SPEC office)
|N1 [ ]||1-year subscription||$ 550||550|
|N2 [ ]||1-year subscription (International)||$ 575||575|
|SPEC Benchmark Suites|
|T3 [ ]||SPEC SFS Release 1.1*||$1200||600|
|T4 [ ]||SPEC CINT92 Release 1.1*||$ 425||218|
|T5 [ ]||SPEC CINT92 Release 1.1*||$ 575||288|
|T7 [ ]||SPEC SDM Release 1.1||$1450||725|
Subtotal _______ _______ 1. Shipping/Handling Charges (see above) _______ _______ 2. Multi-orders discount (see above) _______ _______ 3. Expedited Orders (see above) _______ _______ 4. Sales Tax -- VIRGINIA ONLY (see above) _______ _______ TOTAL _______ _______
2722 Merrilee Drive
Fairfax, Virginia 22031-4499
Phone (703)698-9604, ext. 318 - Rodney Sanford
Managerial and technical inquiries about SPEC should be directed to NCGA at firstname.lastname@example.org.
E-Mail questions that do not interfere too much with our real work :-) can also be mailed to:
email@example.com from North America firstname.lastname@example.org from Europe or elsewhere