Testing is all about testable results, testable properties of programs, right? So it makes sense that we should be able to compare the performance of different testing tools on the same program.
Of course, it isn't that simple. Testing tools are quirky with respect to what kind of software artifacts, source languages, and descriptions they accept. Furthermore, the are designed to tackle different aspects of the testing problem. Not just running and tabulating test cases, but also generating test cases, ordering of test cases, measuring coverage, etc.
Lihua Xu has started working on a testing benchmark, but we've only scratched the surface. Here's a technical report that we wrote together.
Lihua Xu and Susan Elliott Sim, "Towards a Benchmark for Test Generation Techniques," Institute for Software Research, University of California, Irvine, Irvine, CA, USA, Technical Report #UCI-ISR-06-9, June 2006.