Testing patience

Consumer Reports does it again.

After having come in for a great deal of criticism in recent weeks over its AV testing methodology (see

VB, September 2006, p.2

),

Consumer Reports

has – amazingly – damaged its credibility further after having confirmed that, during its testing of anti-spyware applications,

CR

did not test against any spyware.

CR

‘s review of anti-spyware products was based on running the applications against the

Spycar

set of applications that mimic spyware behaviour. While it would be understandable (if not entirely forgivable) that testers lacking experience in the anti-malware field could make such a gaffe, what makes this more astounding is that the creators of

Spycar

state clearly and specifically that

Spycar

is not suitable (as a sole test method) for anti-spyware testing.

The EULA states ‘…Spycar … is intended to be used to see how anti-spyware tools cope with new spyware for which they didn’t have a signature. It is not intended to provide perfect anti-spyware tests, or to act as a substitute for any other form of evaluation.’ And the

Spycar


website

reads: ‘Is Spycar a comprehensive test of anti-spyware tools? No … Spycar does not evaluate the signature base, the user interface, and other vital aspects of an anti-spyware tool. Thus, Spycar alone cannot be used to determine how good or bad an anti-spyware product is.’

After such a controversial performance

CR

will need to work hard if it is to recoup its credibility in the anti-malware field.

Posted on 01 September 2006 by

Virus Bulletin