I received a Cache under-provisioned error.
Happened during a scheduled anti-virus scan.
I guess you can call it a virus scan storm.
Where to go from here? Any suggestions ?
Move away from traditional AV scanning, protect your endpoints and use AV scanning at the hypervisor level is much more efficient and solves your problems.
The IO storms during scans are a very common and there is no solution other than the above, you can mitigate the effect by offsetting the scans. NOTE: this offsetting of the times is something you should also apply to the application of WSUS updates!!
the 2.2 code and higher will prevent random scans and write from flushing the cache. If your not on those code levels, you should upgrade. The other way to work around this is to write a script to disable cache on the effected volume during the scan, and then turn it back on.
We are on 18.104.22.168. code. So we are already on that code level.
If you are using Symantec Endpoint Protection, I would look for a feature called Insight Cache. If you're forced (i.e. compliance) to do 'absolute' FULL scans on every machine every day or week, and your AV scan policies or endpoint groups aren't staggered, I would highly recommend an antivirus solution that compares file hashes on the scanned target, instead of actually scanning each and every file. You might not eliminate all of the load, but it definitely was noticeable for us.
We have Symantec because someone finds it add's value. I could argue that point but I dont.
Instead we run the latest version 12.1.6 (?) the version that allows for a "light" client with drastically reduced definition file sizes and updates. The down side is that it only has definitions for the latest malware. We also have turned off scheduled scans. We only scan on file modification, which for 99% of the files on a VM are never touched after they arrive.
We have lot's of other layers in the environment, PaloAlto, FireEye...etc which actually catch/block stuff.
We also run WSUS updates in the wee hours of the morning.
Retrieving data ...