Open source and the advancement of automated code analysis
Back in January I mentioned Agitar Software in a column on software testing. The backstory was that Agitar got in touch with me after reading my review of Compuware's DevPartner Studio. I had used NLucene, the .NET port of the Java-based Lucene search engine, as a benchmark to explore that product's debugging and source-code analysis features. Agitar's development lead, Kent Mitchell, picked up on the idea. He fed Lucene's Java sources into his test automation tool, Agitator, and used Lucene to demonstrate his product.
Today Agitar's Mark de Visser pointed me to this interesting experiment. It's a set of test coverage reports for Agitar's own product plus some open source Java projects including Ant, Berkeley DB, Cocoon, and Lucene. What exactly these reports mean is open to interpretation, as Agitar points out. Note also that the Agitar is a special case, since the company has been dogfooding its own tool. While "agitation" of arbitrary code can automatically produce a bunch of tests, they're not really mean to be used without human oversight. CTO Alberto Savoia puts it this way:
Agitator can greatly accelerate the development and thoroughness of unit tests by automating most of the activities that don't require human understanding, intelligence, and creativity, but you still need to invest time and thought to direct the automation and to make sure the results the results are correct, robust, and maintainable. [Developer Testing: Eating our own dogfood]
The meta-theme I find interesting here is the virtuous cycle involving open source codebases and a new breed of static and dynamic code analysis tools. Another example: Coverity's Linux bugs database (registration required, see this Linux Magazine article by Benjamin Chelf for background).
To Eric Raymond's famous dictum -- "Given enough eyeballs, all bugs are shallow" -- perhaps we should now add: "Given enough code to study, the eyeballs will be fitted with increasingly powerful spectacles."
Former URL: http://weblog.infoworld.com/udell/2004/06/25.html#a1029
Back in January I mentioned Agitar Software in a column on software testing. The backstory was that Agitar got in touch with me after reading my review of Compuware's DevPartner Studio. I had used NLucene, the .NET port of the Java-based Lucene search engine, as a benchmark to explore that product's debugging and source-code analysis features. Agitar's development lead, Kent Mitchell, picked up on the idea. He fed Lucene's Java sources into his test automation tool, Agitator, and used Lucene to demonstrate his product.