How to scan huge file storage | Kaspersky official blog
Scanning the hard drives of work computers is a simple daily procedure that happens without impacting the user or requiring any manual action. In the case of servers, however, things are more complex — especially if done in response to an incident, after which all company storage (perhaps tens of terabytes worth) need an unscheduled scan. What’s more, you need to ensure absolute data security and no noticeable drop in performance for users.
We’ve compiled a list of tips and precautions to save you time and prevent further incidents. All tips related to our products are using Kaspersky Endpoint Security as an example, but the same logic applies to other EPP/EDR security products.
Preliminary checks
Check the configuration of the computer that will perform the scan. Make sure that the OS is updated to the latest version and can connect to all disks being scanned and process the data correctly — that is: read long Unicode file names, handle very large files and files on case-sensitive partitions, and so on. To speed up the scan, use a computer with a powerful multicore CPU, generous memory, and fast local storage for temporary files.
Make sure that disk-access is fast. The computer should connect to all storage either directly (local storage) or through a fast network interface using a high-performance protocol (preferably SAN-type).
Check your backups. Although scanning should not affect stored data, it’s important to have a plan B in case of malware infection or file corruption. Therefore, carefully check the date and contents of the most recent backup of all data, consider when data-recovery drills were last performed, and generally make sure the current backup versions are usable. If current backups aren’t available, assess the risks and time frames, and possibly back up critical data before scanning.
Clarify the nature of the data on the disks and the storage specifications. This is to optimize the scan settings. Are the disks arranged in a RAID array? If so, what type? You need to decide whether to scan different disks in parallel, and whether this will boost performance. If the disks are accessible independently, consider parallel scanning from multiple computers. Here again, both access speed and server capacity are key. For a powerful computer limited mainly by access speed to different disks, you can run parallel scanning tasks on a single machine.
The nature of the data will greatly affect your decision. If the disks contain many heterogeneous files, or archives with a large number of files, scanning will require significant resources of all types: CPU, memory, temporary folders, etc. The load will be lower if large files in a safe format (video editing sources, database tables, backups/archives known to be untouched) make up a major part of what’s being stored.
Preparing for scanning
Schedule the scan time. Ideally, a weekend, nighttime, or other period when few users access the data. Then you can either completely remove the disks and servers to be scanned from public access, or warn users about possible system slowdown and be sure that only a very small group of people will be affected.
Make sure there’s enough free space on the disks. Scanning may involve unpacking archives and images, which sometimes requires a lot of space.
Check quarantine storage settings. If many infected and suspicious files are found, quarantine may overflow and older samples will be deleted. So it’s worth allocating plenty of space for quarantine.
Agree and enforce an exclusion policy. To reduce scan time, exclude resources that pose no risk and would take a very long time to scan. This category typically includes very large files (with the cutoff ranging from hundreds of megabytes to several gigabytes, depending on the situation), distribution kits, backups, other files that haven’t been modified since previous scans, and files that are known to be non-executable. However, the last category is not so clear-cut, as there can be malicious fragments hidden in plain text files and images. So it’s better to be safe than sorry and scan images as well.
Delete temporary files and folders so you don’t waste time on them.
Scan settings
These recommendations should be adjusted in line with your prior assessments and the nature of the data, but the basic advice is:
- Set the maximum amount of memory and CPU time for scanning, taking into account the server usage profile. If the server is unavailable to users during scanning, you can allocate up to 80% of CPU and memory resources — any higher and the computer may become sluggish. For servers that remain under normal load, these numbers should be significantly lower.
- In our product settings enable iChecker and iSwift. These technologies speed up scanning of some file formats and exclude data that’s been unchanged since the last scan.
- Here, you can also enable additional options to prevent overloading the system: ” Do not run multiple scan tasks at the same time” and “Scan only new and modified files”.
- Disable scanning of password-protected archives; otherwise, password requests will cause the application to stop scanning.
- Set the maximum size of files for scanning in accordance with what we discussed above.
- Set the heuristic analysis level to medium.
- Select actions for infected objects; quarantine will likely be the best choice.
- Set the logging settings so that the logs contain sufficiently detailed information about scanned objects and scan results.
Performance settings are described in more detail on our support site: for Windows and for Linux.
Running the scan
Start by scanning a small partition or subset of files weighing no more than a terabyte. Evaluate the impact of the scan on server performance (especially important if it continues to serve users) as well as the total time taken, and check the logs for errors. If the scan seems to take too long, try to figure out from the logs what caused the bottleneck. Using this data, adjust the settings accordingly and schedule a “big scan”.
Even after the test, we don’t advise running a full scan of the entire data volume in one task. It’s better to create multiple scan tasks — each targeting only one of the many storage fragments, such as individual disks. This reduces the risk of a prohibitively long scan time, or a failed scan that has to be restarted from scratch.
In the basic scenario, these subtasks are run sequentially as they’re completed. But if the system configuration allows it, dividing the scan into multiple tasks will let you scan independent disks in parallel.
During scanning, monitor the system load and the scan progress so as to intervene in time in case of abnormal situations. And after each task is completed, be sure to drill down into the logs!
Kaspersky official blog – Read More