Benchmark Definitions (Computing)

In computing, a benchmark is the result of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, by running a number of standard tests and trials against it. - From Wikipedia

A Benchmark is a standardized program (or detailed specifications of program) designed or selected to be run on different computer systems, with the goal of a fair comparison of the performance of those systems. - From R. Weicker

Grid Benchmark Definitions

The use of benchmark programs for the fair, concise, and affordable performance characterization of different aspects of a Grid infrastructure. - From M. Dikaiakos

A result of running a set of standardized programs to fairly assess the performance and dependability of different aspects of Grid infrastructures, services and applications. - From me

Performance-based Grid Benchmarking Tools

Benchmarking Grid Infrastructures

• GridBench: A Tool For Benchmarking Grids - Operation/s for CPU performance, Available Memory and Bytes/s for memory/cache, Latency and Bandwith for interconnects
• SPECint2000 and SPECfloat2000 - CPU performance
• NAS Grid Benchmarks: GridNPB 3 - Turnaround time, Accumulated CPU Time, Throughput
• Grid Assessment Probes (GRASP) - Execution times, Failure

QoS-based Benchmarking Grid Software

• VegaBench: A Benchmark Tool for Grid System Software - I do not have this paper

People/Groups Researching in Performance-based Grid Benchmarking

• Grid Benchmarking Research Group - Global Grid Forum
• GridBench: A Tool For Benchmarking Grids - Dept. of Computer Science, University of Cyprus
• IPACS Benchmark - Integrated Performance Analysis of Computer Systems
• SPEC - Providing SPECint2000 and SPECfloat2000 for MDS to benchmark CPU performance
• IETF Benchmarking Methodology (BMWG) - The Internet Engineering Task Force
• Marios D. Dikaiskos - Dept. of Computer Science, University of Cyprus
• George Tsouloupas - Dept. of Computer Science, University of Cyprus
• William Gropp - Senior Computer Scientist, Mathematics and Computer Science, Argonne National Laboratory
• Rob F. Van der Wijngaart - Computer Sciences Corporation, NASA Ames Research Center
• Allan Snavely - PMaC Laboratory Director, San Diego Supercomputer Center, Department of Computer Science and Engineering, University of California, San Diego

Performance-based Grid Benchmarking Presentations

• Grid Benchmarking: Why Implies How - William Gropp, Mathematics and Computer Science, Argonne National Laboratory
• Grid Benchmarking - George Tsouloupas, Marios D. Dikaiskos, Pedro Trancoso
• Design and Implementation of GridBench - George Tsouloupas, European Grid Conference, Amsterdam, February 14th, 2005
• Grid Performance, Grid Benchmarks, Grid Metrics - Zsolt Nemeth, MTA SZTAKI Computer and Automantion Research Insitute

Performance-based Grid Benchmarking Papers

• NAS Grid Benchmarks: A Tool for Grid Space Exploration - Michael Frumkin, Rob F. Van der Wijngaart, In Proceedings of the HPDC01, Page: 03 15, 2001
• Evaluating the Information Power Grid Using the NAS Grid Benchmarks - Rob F. Van der Wijngaart, Michael A. Frumkin, 18th International Parallel and Distributed Processing Symposium (IPDPS'04) - Workshop 17, 2004
• Benchmark Performance on Cluster Grid with NGB - Liang Peng, Simon See, Jie Song, Appie Stoelwinder, Hoon Kang Neo, 18th International Parallel and Distributed Processing Symposium (IPDPS'04) - Workshop 17, 2004
• Benchmark Probes for Grid Assessment - Greg Chun, Holly Dail, Henri Casanova and Allan Snavely, 18th International Parallel and Distributed Processing Symposium (IPDPS'04) - Workshop 17, 2004
• GridBench: A Tool for Benchmarking Grids - G. Tsouloupas, M. Dikaiakos, In Proceedings of the 4th International Workshop on Grid Computing (Grid2003), pages 60-67, Phoenix, Arizona, 17 November 2003, IEEE Computer Society
• Ranking and Performance Exploration of Grid Infrastructures: An Interactive Approach - G. Tsouloupas and M. D. Dikaiakos, Proceedings of the 7th IEEE/ACM International Conference on Grid Computing, p. 313-315, Barcelona, September 28th-29th 2006, IEEE Computer Society
• Grid Benchmarking: Vision, Challenges, and Current Status - M. D. Dikaiakos, Concurrency and Computation: Practice and Experience Journal, Wiley. June 13, 2006.
• A Performance Study of Monitoring and Information Services for Distributed Systems - X. Zhang, J. Freschl, and J. Schopf. Proceedings of HPDC2003, August 2003
• Understanding Grid Resource Information Management through a Synthetic Database Benchmark/Workload - Beth Plale, Craig Jacobs, Scott Jensen, Ying Liu, Charlie Moad, Rupali Parab and Prajakta Vaidya, CCGrid04, 2004
• VegaBench: A Benchmark Tool for Grid System Software - H. Zhu, Y. Zou, L, Zha, Fifth International Conference on Grid and Cooperative Computing Workshops pp. 543-548, 2006
• A Benchmark Suite for SOAP-based Communication in Grid Web Services - Michael R. Head, Madhusudhan Govindaraju, Aleksander Slominski, Pu Liu,Nayef Abu-Ghazaleh, Robert van Engelen, Kenneth Chiu, Michael J. Lewis, SuperComputing05, November 12-18, 2005, Seattle,Washington, USA
• Benchmark Evaluation of Xindice as a Grid Information Server - Prajakta Vaidya Beth Plale Technical Report TR585, Computer Science, Indiana University, 2003
• An Evaluation of Alternative Designs for a Grid Information Service - Warren Smitht, Abdul Waheed*, David Meyers', Jerry Yan, In IEEE HPDC00, August 2000.

Useful Information

• Analysis of Benchmark Characteristics and Benchmark Performance Prediction - Rafael H. Saavedra and Alan J. Smith, ACM Transactions on Computer Systems (TOCS) 1996
• The role of performance engineering techniques in the context of the Grid - Tony Hey, J. Papay, M. Surridge. Concurrency and Computation: Practice and Experience, Volume 17, Issue 2-4 , Pages 297 - 316, 2005

Dependability-based Grid Benchmarking

Tools

• Grid-FIT - a Fault Injector that allows network level fault injection to be used to test SOAP based Grid systems
• FAIL-FCI - Fault Injection Language, (Not available now)

Papers

• An overview of existing tools for fault-injection and dependability benchmarking in grids - Sebastien Tixeuil and William Hoarau and Luis Silva. Technical Report 0041, CoreGRID, October 2006