Testing bodies AV‐Comparatives, AV‐TEST and Virus Bulletin comment on allegations of inappropriate behavior Today, three of the world’s most renowned and trusted security testing bodies, AV‐Comparatives, AVTEST and Virus Bulletin, stand united to censure security vendor Qihoo 360 after finding the firm submitted products for comparative and certification testing which behaved significantly differently from those made available to its users and customers. The three testing bodies will revoke all certifications and rankings awarded to the company’s products so far this year, and going forward will insist on more open and fair dealings to ensure users are provided with the most accurate information possible. Investigations by the three labs found that all products submitted for testing by Qihoo had one of the product’s four available engines, provided by Bitdefender, enabled by default, while a second, Qihoo’s own QVM engine, was never enabled. This included versions posted to ostensibly public sections of the company’s websites. By contrast, as far as can be determined, all versions made generally available to users in Qihoo’s main market regions had the Bitdefender engine disabled and the QVM engine active. According to all test data this would provide a considerably lower level of protection and a higher likelihood of false positives. Options are provided in the product to adjust these settings, but as the majority of users leave settings unchanged, most tests insist on using the default product settings to best represent real‐world usage. As part of the investigation into Qihoo 360, counter‐accusations were levelled by the company against two fellow Chinese security firms, Baidu and Tencent. Analysis of products submitted for testing by these companies turned up some unexpected flags within their products, marked with the names of several test labs and implying some difference in product behavior depending on the environment they were run in – similar flags were also found in Qihoo products. However, no evidence could be found that this gave any significant advantage to either product, and in some cases it even seemed to put them at a disadvantage. Both firms were able to provide good reasons for including these flags in their products.
On requesting an explanation from Qihoo 360 for their actions, the firm confirmed that some settings had been adjusted for testing, including enabling detection of types of files such as keygens and cracked software, and directing cloud lookups to servers located closer to the test labs. After several requests for specific information on the use of third‐party engines, it was eventually confirmed that the engine configuration submitted for testing differed from that available by default to users. Qihoo’s awards and certifications attained since the start of 2015 will thus be stricken from the records by all three testing bodies, and all three will be imposing stricter demands on test participants to avoid any further gaming of results by vendors. “This sort of thing doesn’t really help anyone,” said John Hawes, Chief of Operations at Virus Bulletin [[email protected]]. “Independent tests serve both users and developers, showing which products are performing best and highlighting areas where developers need to work harder. If the products being tested aren’t those being used in the real world, nobody’s getting any useful information.” Andreas Clementi, CEO of AV‐Comparatives [[email protected]], said: “Independent antimalware testing plays a key role in raising the standard of protection on users’ devices, which in turn makes the Internet a safer place for everybody. Misuse of such tests for marketing purposes will, in the long run, result in more successful malware attacks, making Internet users less secure.” Maik Morgenstern, CEO of AV‐TEST [[email protected]], said: “Comparative testing and certification plays an important role in the anti‐malware industry, both for the users and the vendors. Users rely on independent results to make an educated decision regarding their protection software. If vendors start to manipulate the testing process, they are hurting everyone involved.”
About AV‐Comparatives AV‐Comparatives [av‐comparatives.org] is an independent organization offering systematic testing that checks whether security software, such as PC/Mac‐based anti‐virus products and mobile security solutions, lives up to its promises. Using one of the largest sample collections worldwide, it creates a real‐world environment for truly accurate testing. AV‐Comparatives offers freely accessible results to individuals, news organizations and scientific institutions. Certification by AV‐Comparatives provides an official seal of approval for software performance which is globally recognized. Currently, AVComparatives’ Real‐World Protection Test is the most comprehensive and complex test available when it comes to evaluating the real‐life protection capabilities of anti‐virus software. Put simply, the test framework replicates the scenario of an everyday user in an everyday online environment – the typical situation that most of us experience when using a computer with an Internet connection. AVComparatives works closely with several academic institutions, especially the University of Innsbruck’s Department of Computer Science, to provide innovative scientific testing methods.
About AV‐TEST AV‐TEST [av‐test.org] GmbH is an independent supplier of services in the fields of IT security and antivirus research, focusing on the detection and analysis of the latest malicious software and its use in comprehensive comparative testing of security products. Due to the timeliness of the testing data, malware can instantly be analyzed and categorized, trends within virus development can be detected early, and IT‐security solutions can be tested and certified. The AV‐TEST Institute’s results provide an exclusive basis of information, helping vendors to optimize their products, special interest magazines to publish research data, and end‐users to make good product choices. AV‐TEST has operated out of Magdeburg (Germany) since 2004 and employs more than 30 team members, professionals with extensive practical experience. The AV‐TEST laboratories include 300 client and server systems, where more than 1,000 terabytes of independently collected test data, containing both malicious and harmless sample information, are stored and processed.
About Virus Bulletin Virus Bulletin [virusbtn.com] is an online security information portal and certification body providing users with independent intelligence about the latest developments in the threat landscape, as well as conducting bimonthly certifications of anti‐malware and anti‐spam products. Both its VB100 antimalware tests and VBSpam spam filter tests are well recognised and highly respected in their fields. Virus Bulletin also organises the VB Conference, an annual event at which the brains of IT security from around the world gather to learn, debate, pass on their knowledge and move the industry forward – the 25th Virus Bulletin Conference will take place in Prague 30 September to 2 October 2015. Virus Bulletin is supported by an Advisory Board comprising some of the world’s leading anti‐threat experts.