How we test antivirus software?
Testing antivirus software is much more complicated than checking out a word processor or a web browser. Antivirus programs must stay on top of the latest and most insidious threats in real time, and it's basically a cat-and-mouse game between the developers and the hackers.
When testing the security of our systems, we can't just rely on static menus and feature checklists. We have to simulate the real world, where malware can attack at any moment, in a safe, controlled environment.
Introduction
Antivirus testing presents a very different scenario from traditional software testing. The goal is to see how well the product can detect, block, and eliminate malware without causing problems with the rest of the system.
Another layer of complexity comes from the dynamic nature of malware, such as trojans, ransomware, and spyware, which seem to be popping up everywhere, with brand-new ones appearing every day and using tricks to avoid being detected. To test an antivirus system, you can't rely on the same old virus samples or run-of-the-mill test files; you need something a lot more up-to-date. This is why we at PCRisk always have the latest, real-world malware samples in our test arsenal so that we can see how the software can handle something it's never seen before.
It's also rather dangerous, since we are releasing live malware into our lab, so we've developed a series of high-tech safety measures to prevent the malware from spreading and getting out of control. What we're looking for in an antivirus is a complete combination of protection, precision, performance, and user-friendliness. In the following sections, we'll outline how we at PCRisk approach antivirus software testing.
Why You Should Trust PCRisk's Expertise
With respect to evaluating cybersecurity software, PCRisk stands out from the rest. Every day, our team of malware experts faces the very threats antivirus software is trying to combat. They're not just isolated in a lab; they're right on the frontlines, cutting through the latest and most pernicious malware that's infecting computers.
Our team's experience with adware infestations, spyware, and particularly nasty ransomware outbreaks provides a clear picture of the types of threats spreading in the wild. We enter the risky realm of questionable websites, commonly linked to exploits and phishing schemes, and we pick up the malicious files and links they spread and add them to our malware library.
We get thousands of nasty emails, infected attachments, and phishing attempts every week. All of them feed into our database, so when we test an antivirus program, we send it brand-new, bleeding-edge malware that hackers are currently using. Not just stale old test files or simulations. We can therefore get an accurate measure of how any given antivirus will cut loose in real-world threats.
Our years of malware analysis and background in the field also mean we can verify what's being caught and what's being missed. Since our experts know exactly what makes malware tick, we can tell instantly when an antivirus takes down an attack and when one slips past its defences. Lots of review sites can do basic feature checks and run pre-programmed performance tests, but we go way beyond that.
When testing antivirus software, we simulate real-world attacks that regular people face every day. With our in-depth knowledge of malware and our years of experience in the field, we're able to see how each product performs under the worst-case scenario. We don't just test software; we recommend it when we know that it's capable of withstanding the kind of attacks that hackers can launch. Our hard-earned expertise in fighting against malware forms the basis of every review we put out to the public.
How We Test Malware Detection Capabilities
When evaluating antivirus programs, the primary test is to see how well they can kill malware, which is what PCRisk focuses on (the first and most important aspect). The company's test for malware detection has been set up to allow the program to catch any threats before they can cause harm.
Coming from different angles, the test includes checking the antivirus's real-time protection, mimicking how malware typically gets into a system, through downloads, email attachments, and infected USB drives. We run the antivirus in the background and see if it's able to flag or block the harmful files as they hit the system. A top-notch antivirus should absolutely crush the execution of any malware on the spot, and ideally before it even gets a chance to install or run.
We record what samples are stopped by the real-time shields and note how the program informs the user that a threat has been neutralised. Next, we test on-demand scanning by doing full system scans and custom scans, and since we don't want the real-time protection to delete our test samples, we disable it or get past it. Notorious malware samples and ones in weird places all over the system are seeded onto the test system, and then we run a complete scan and measure how many of these nasty samples the antivirus can catch.
Our test library has an impressive array of malware, including trojans, worms, ransomware, rootkits, keyloggers, and more, and some of these samples are brand-new, still unknown to many other security programs. We review the results, checking what threats were found, how they were classified, and what the antivirus suggested to be done about them. We use a combination of real-world and controlled conditions to measure its ability to detect a wide range of malware.
However, detection is not the end of the story. We also monitor the capabilities of the antivirus to remediate and remove malware. We use a scale to measure the steps the software takes after an infected file is detected. Whether or not the file is automatically deleted or quarantined is only one measure, but clean remnant traces in memory, registry entries, and other malicious changes are also evaluated. In cases where we introduce an infected file, kill it and see whether our scans contain or get rid of the virus's malicious operations, restore any alterations the virus made, and basically return the system to its state before the infection. We also run a second scan after the cleanup to verify that no nasty bits are left behind.
We meticulously track each sample and don't just look at the binary results. We also factor in the time it takes to run the scan, and if the antivirus software struggles with certain types of malware. From the user perspective, we're interested in whether the alerts and menu options are clear when a threat is detected. Essentially, we're looking for clarity and usability, in addition to protection. Our testing is thorough and demanding for the antivirus engine, pummeling it with a wide array of real-world and controlled threats. We then evaluate not only how many threats the antivirus identifies, but how effectively it eliminates those threats, all the way down to the cleanup.
Usability and Interface Testing
When it comes to a security program, malware detection capabilities are essential. Still, it won't be effective if the user can't figure out how to use it, nor will an antivirus be if it's clunky to install and slows down your computer. PCRisk's rating system is designed to test the usability and interface of every antivirus we review. We check how easy it is to install, how user-friendly the interface is, and what kind of hit it takes on your system's performance. We want to ensure that an antivirus keeps you safe and doesn't make you want to get rid of it.
We start with the installation and setup experience. It's essential to have a seamless experience, and the download and installation of the software should be smooth and clear, with the installation instructions to follow being to the point. Any bundled software or toolbars is something we dislike and the installation should be a 5-10 minute affair, involving a few steps before the antivirus is operational.
We don't want an antivirus that asks for overly technical decisions, such as setting up advanced settings or creating a user account, because these can be confusing to non-technical people. We're looking for a quick and straightforward installation that doesn't require a genius IQ. We also note if a reboot is necessary after the installation and if the software starts running a full scan or updates automatically, as these things can affect the user's initial impression.
We take the time to thoroughly understand its user interface, or UI, when evaluating an antivirus. Coming hotfooting into the main dashboard and menu systems, we look to see how information is laid out and presented, and if there are any red flags in how important information is displayed. A user-friendly antivirus will show us if we're protected, when the last scan was run, and if any issues require attention without forcing us to dig through complicated menus. We check the location of key features such as scan, update, quarantine, and settings, and look for simplicity in the settings panels.
The best UIs will also provide both the novice and the power user with what they're looking for: novice users need clear and simple options, possibly a one-click mode or recommended settings, and advanced users want to be able to tweak configurations, and so on. We're also watching for any jarring language or visual clutter, confusing labels, and descriptions that don't add clarity. The look and feel of an antivirus can be the deciding factor in a user's comfort, so we consider the legibility of text, the variety of icons, and the general aesthetic of the software.
As for usability, we also consider the way antivirus software handles notifications and alerts. We check the frequency and clarity of pop-up messages and do not want the software to bombard the user with a deluge of constant alerts or pummel them with requests to purchase a premium version, as is the case for some free antivirus tools.
Regarding notifications, we go for the "less is more" approach. Requiring the user's attention only when necessary, such as when a threat is detected or an update is needed. When a notification does pop up, we want to know exactly what's going on and be able to tell the user what happened, in clear, non-technical language, and what action was taken. Coming hotfooting back to ask the user what to do with a suspicious file should be avoided, so we lay out the options in a way that's easy to get through and steer the user towards the right decision. We also test the antivirus as an ordinary customer would, for a time, to see if there are any quirks or confusing parts.
When assessing our antivirus's performance, we examine its impact on system performance since background running security software can be a major annoyance. We measured the performance of our antivirus in this aspect by running subjective tests. We get on with our normal computer activities, opening programs, browsing the web, copying files, and playing media. We note if the antivirus is causing any slowdowns or sluggishness that weren't there before.
One of the things we do is run a series of basic performance benchmarks and routine tasks with and without the antivirus turned on when testing an antivirus. We measure how long it takes to boot into the desktop, or how quickly a standard set of applications launches, and we look at CPU and memory usage. Furthermore, we look at whether an antivirus consumes lots of RAM, and/or sends the processor into overdrive, even when the computer isn't doing much else. We also see how heavy the load is and how long a full system scan takes.
We check if the antivirus is intelligent and will pause or slow down when we're actively using the PC. We also look for features like a "game mode" or a scheduler that lets you run scans at night.
In summary, when assessing the usability and interface of an antivirus, we want to know that the software not only has the power to protect against malware but also empowers the user, giving them a sense of security without the hassle and confusion while staying practical. We go one step further and install the antivirus exactly as a regular user would and live with it for a period of time. We're confident that a top-rated antivirus on PCRisk will merge impressive protective capabilities with a user-friendly experience.
Testing Additional Features
Looking at modern antivirus software, you'll often find a whole suite of features beyond the basic virus scan, and while these can be extremely helpful, it's up to us to verify that they are genuinely effective. PCRisk puts each major feature in an antivirus package under the microscope as part of our thorough review process. We look for features that can stop phishing attacks, block annoying adware, manage passwords, filter out malware, perform a rootkit scan, control webcam use, and much more. Some of the key features we evaluate include:
Ransomware Protection: With respect to ransomware, many antivirus programs on the market now have a specific feature to combat ransomware. We test these features in a lab setting, simulating a real ransomware attack and seeing if the software's monitoring can detect and stop the encryption. We unleash an actual ransomware sample that we've saved up in our collection, watching its behavior to see if the software can intervene in time.
We look out for how fast the ransomware is stopped and if any files are encrypted before the threat is neutralised. Some antivirus programs come with a rollback feature, essentially restoring files or changes made by the ransomware. We see how effectively this feature works and if it can recover what was lost. We aim to ensure that the anti-ransomware feature is not just a label but capable of fighting against ransomware, no matter if it's a brand-new strain.
Firewall and Network Defense: When evaluating a security suite, we check for a built-in firewall or network protection module to see if it effectively guards against unauthorized access and network-based attacks. We perform various network activities that a personal firewall should be able to block, and then test port scanning on our test system from another machine to see how they get handled. Then, we run unknown or suspicious software on our test system, and check to see if the firewall will send them to limbo, block them out, or ask for our approval.
A top-notch firewall will, of course, shield against incoming attacks, but also manage outgoing communication, ensuring that your malware doesn't "phone home" with your data, and take note of how user-friendly the firewall is. Since intrusion detection and Wi-Fi network scanning features are often part of a security suite, we perform simple tests to see if they do their job.
Browser and Web Protection: When discussing blocking malware and phishing attacks, antivirus suites usually have a web-based component in the form of a browser extension or web filtering feature. We verify the effectiveness of web protection by visiting a list of known malicious URLs. The websites we visit are, for instance, phishing sites designed to steal login credentials, pages harbouring drive-by download exploits, and URLs that lead directly to malware downloads. Our experiments are conducted in a separate environment.
We test whether an antivirus can block access to the malicious sites outright or intercept the download before it does any harm. We're also looking at whether this protection is cross-browser, works with Chrome, Firefox, Edge, etc., or is limited to one. The web protection shouldn't interfere with normal browsing, only flagging dangerous sites, and allowing safe ones to load as they normally would. Since adequate web protection virtually eliminates the possibility of being exposed to malware, web protection is one of the main features we consider when rating suites with this capability.
Email Scanning and Antispam: When testing antivirus software, we look for email protection. Unfortunately, malicious emails with attachments and phishing links are all too common, so many of our antivirus solutions have built-in email scanning or spam filtering features. We test email protection by sending some of the nasty attachments we've caught in our honeypots to an email client protected by the antivirus. In close pursuit of cybercriminal tactics, we've employed macro-infected documents, malware-laden fake invoices, and trojan-filled compressed archives.
We try to open or save these attachments to see if the antivirus springs into action, detecting the threat as soon as the file is accessed. And if the product has an anti-spam component, we can see how well it can sort out the rubbish from legitimate emails in the inbox, sending a batch of test emails with known spam signatures and phishing content to check its accuracy. We're also ensuring the email scanner doesn't ruin the normal mail flow. If it can do all this, the antivirus essentially serves as a last line of defence against nasty email threats, bailing out whatever spam filter the email service is already running.
Bundled Tools (VPN, Password Manager, etc.): Regarding security suites, many offer additional tools, all packaged under one roof, to provide a complete solution for users. As you explore a typical security suite, you'll find VPN services for secure browsing, password managers that can securely store login credentials, system optimization tools, file shredders or encryption software, parental controls, and more. Although these aren't the core of malware protection, they add value to the product. At PCRisk, we don't disregard these features, and in our testing, we make sure to try them all to see if they're functional and bring real value to the table.
When we evaluate suites, we look closely at the software that comes with them. If the suite includes a VPN, we'll set it up, hook it up to a server, and confirm that it's effectively masking our IP and keeping our speed reasonable. Password managers are tested for creating and auto-filling passwords to see how user-friendly and secure they are, and if they require a master password, and use top-notch encryption. System clean-out tools are run to check if they're really clearing out temp files, or just mimicking the work of the built-in cleaners. We aim to see whether these add-ons are fully functional and beneficial or just a gimmick. A feature shouldn't be half-baked; it needs to be able to hold its own against the standalone apps, and we're looking out for any of these extras that require extra cash or have limitations within the suite. For instance, some "free" VPNs will hit a data cap unless you stump up more money.
When testing additional features in the antivirus, we're also ensuring our reviews give a complete picture. We expect all the components that are part of a full security suite to be up to the mark. We don't just take the word of the manufacturers; we put each claim to the test. If they claim to protect against ransomware, we feed them real ransomware; if they have a firewall, we launch network attacks at it. This is basically the only way to know that the whole package is top-notch. Conversely, if there are weak points in any extra features, we'll identify them so users know exactly what they're getting.
What Hardware We Use
We want a state-of-the-art setup that can handle live malware and performance analysis when conducting antivirus tests. Here at PCRisk, we've built a highly adaptable laboratory environment equipped to perform these tests in a controlled and efficient manner. Utilizing a combination of physical and virtual test machines, we can cover all of our bases. The ability of our virtual machines to create "clean snapshots" of an operating system can be a lifesaver in our line-up of tests. We set up a brand-new virtual machine each time we review an antivirus.
Think of a Windows 10 or Windows 11 installation straight from the box, with normal settings and typical software. We then snap a picture of this pristine state of the system so we can go back to it after testing and prevent any cross-contamination from one test to the next. Our virtual machines are configured with fairly standard specs, like a mid-range multi-core processor, 4-8 GB of RAM, and loads of hard drive space, mimicking the average user's PC.
Our virtual machines allow us to safely run malware. If the virus causes serious damage or knocks the system out, we can revert to the clean snapshot in seconds. Plus, we isolate these VMs so they don't get connected to our main network or the Internet unless we want them to, such as when testing cloud-based features.
Fighting against anti-VM malware malicious programs that know they're running in a virtual machine and then change their behaviour or won't run. We have to get a bit more aggressive. Popular methods include changing the virtual machine's settings and surroundings, basically hiding the signs that say it's running in a virtual environment. Still, the best way to outsmart them is to use physical test machines, which we do when necessary. Reboot Restore is an excellent tool in that scenario, as it allows us to rapidly restore a physical machine to its pristine state after we've finished testing.
To test the security of our systems, our lab is equipped with several physical PCs, ranging from high-performance rigs to older, lower-spec machines. Physical hardware is very important in this line of work, as some advanced malware will only run its payload on real hardware. That's why a physical machine is the only way we can completely observe and test how antivirus software manages and destroys these types of threats.
Secondly, performance testing can be more accurate on physical hardware - we can see precisely how an antivirus knocks around with system startup, memory use, and app performance on a genuine machine, minus the tiny bit of overhead that virtualization brings. We're basically setting up different hardware profiles, too. One is a brand-new desktop with a lightning-fast SSD and loads of RAM, so we can see how the antivirus performs when there's no shortage of resources. The other is an old laptop or a cut-price PC, where we check how efficiently the software runs on a tighter system.
Our virtual and physical systems are equipped with genuine, updated operating systems and standard software. We add real user settings and stock the systems with mock personal files, including documents, pictures, etc., to make the environment feel all too familiar and real. This allows us to gauge the effectiveness of our malware and its ability to monitor performance.
Since internet access is often required for some test features, we hook our test systems up to a controlled gateway. This gateway keeps a close eye on the network activity and will block anything that tries to sully the system. Our setup is designed so we can unleash real-world threats, see the antivirus at work, and then, if need be, wipe the slate clean and do it all over again.
We use a combination of virtual and physical machines, meaning we can view the antivirus software from any angle. If it performs well on a virtual machine but lets down real PCs, we'll catch that, too. If a piece of malware is smart about virtualisation, we'll get past it by running the test on real hardware. It's down to the smallest details that give us the reliable results we need.
How We Score Antivirus Software
After we complete the series of tests for an antivirus product, the final step is distilling all those results into a clear evaluation. PCRisk uses a weighted scoring system to rate antivirus software across several crucial categories. When evaluating software, we look at different aspects of its performance, each with varying degrees of importance to the user. We apply weights to these categories to give the final score a fair representation of what matters, such as malware detection, and still account for secondary features and user interfaces. Our scoring criteria is broken down into 4 main categories, each with a specific weight.
1. Malware Detection and Removal (60%) - The ability to neutralize malware takes center stage for the score. It includes detecting and eliminating threats like viruses, trojans, spyware, ransomware, and the rest of the pack. Our real-time protection tests and one-off scans feed into this section, and the result is that a product that nabs nearly all the malware and thoroughly cleans out existing infections gets high marks, but those that don't get a lower score. We've given this area more than half of the total score because, if an antivirus doesn't protect against malware, it's basically failing its job.
2. Performance Impact (15%) - We look at its impact on the system's speed and responsiveness when measuring the performance of an antivirus software. We score the software on efficiency by how light it runs on a system. If we observe significant slowdowns, high resource consumption, or anything detrimental to everyday use, such as long delays opening programs, or very slow scan times that have no option to reduce the impact, we deduct points. Coming in at 15% of the total score, the impact on performance is a huge consideration. An antivirus that cripples a computer will cause users to disable or uninstall it, and in essence, render the protection useless.
3. Usability and Interface (10%) - We consider the ease of use and user interface. When we rank antivirus software, we consider how easily a normal person can install, navigate, and get the most out of the product. Criteria we look for include an intuitive interface, clear settings, minimal ads and pop-ups, helpful alerts, and a silky-smooth experience. If an antivirus is confusing, has a clunky interface, or is a real pest when it comes to asking for more money, its score takes a hit. At about 10% of the total score, a product's usability can be the deciding factor between two antivirus programs offering similar protection.
4. Feature Set and Extras (15%) - Looking at the rest of the score, the value of the antivirus suite's additional features takes centre stage. We examine what else the antivirus offers, beyond just the basic scanning capabilities. A complete suite might have a firewall, password manager, secure web browser, file shredder, system cleaner, parental controls, and identity theft protection, and it's not just about whether these features are there, but also how well they perform in our tests. We give top marks to products that come with useful and functioning bonus features that either add to the security or privacy of the user. For instance, a strong ransomware rollback feature or a cracking good VPN service would score very well here. However, if a product is almost empty-handed or its additional features didn't cut it, it'll score poorly. This feature section weighs in with 15% of the total score.
We use a multi-faceted system that incorporates various components to generate a score, and we believe this approach is fair and open when reviewing antivirus software. The most important function is the level of malware protection, and even if a product isn't the best in every aspect, if it's killing the competition in malware protection, it will score well with us. Well-known products stuffed to the brim with features, but are just average at protection, won't be receiving top marks, because protection matters most to users. We also believe that our weighted approach aligns with what people expect from security software. Our explanations of the strengths and weaknesses of a product flesh out the details, and the final score shows how the software ranks.
The numbers are not set in stone, however. If the software improves, we can re-score it. Fixing problems or improving detection, and if the threat landscape changes, we'll re-run the tests when we score and review antivirus software. We'll update our reviews and our "best of" lists based on the latest results, which is why you can count on a top-rated product on PCRisk having been thoroughly tested and proven itself in the areas we've mentioned.
When we say that a particular antivirus is the best, it's because we've put it through a rigorous testing process that includes pokes and prods from every direction. What we're left with is an entirely accurate measure of its capabilities. The hard work we do in the lab is designed to get to the heart of what's best for your digital life.