MALWARE ANALYSIS EVASION TECHNIQUES AND LIMITATIONS OF MODERN DETECTION TACTICS
Authors: Lui Jangulashvili
Affiliation: Ilia State University, LLC Orient Logic
Category:
Keywords: Malware analysis, evasion techniques, obfuscation, dynamic analysis, machine learning, sandbox detection
ABSTRACT. Malware analysis is one of the critical domains of cybersecurity, since malware is constantly evolving, new techniques and tactics are developed by adversaries. Several different methods and types of malware detection are present in defensive security specialists’ tool set, but evasion techniques are just as vast and diverse. Following paper tries to thoroughly scrutinize different techniques malware authors utilize to circumvent analysis, and categorize them based on type of defensive methods they are trying to bypass. Research explores common forms of code obfuscation used to dodge detection by automatic and static analysis, Methods for detecting VM, sandbox or debugger during manual dynamic analysis and how recently developed machine learning solutions are applied in defensive context. Lastly, paper concludes what are the main limitations of contemporary solutions and highlights remaining vulnerabilities and blind-spots in modern defensive approaches
References:
AV-Test. The independant IT security institute. 2024, URL https://www.avtest.org/en/statistics/malware/.
Amir Afianian, Salman Niksefat, Babak Sadeghiyan, and David Baptiste. 2019. “Malware Dynamic Analysis Evasion Techniques: A Survey” APA Research Center, Amirkabir University of Technology
Matthew Gaber, Mohiuddin Ahmed and Helge Janicke. 2025. “Defeating evasive malware with Peekaboo: Extracting authentic malware behavior with dynamic binary instrumentation”. Journal of Information Security and Applications 95 (2025) 104290
Ilsun You and Kangbin Yim. 2010. “Malware Obfuscation Techniques: A Brief Survey”. Proceedings of the Fifth International Conference on Broadband and Wireless Computing, Communication and Applications, BWCCA 2010, November 4-6, 2010, Fukuoka Institute of Technology, Fukuoka, Japan (In conjunction with the 3PGCIC-2010 International Conference)
Daniel Gibert, Carles Mateu, Jordi Planes and Joao Marques-Silva. 2020. “Auditing static machine learning anti-Malware tools against metamorphic attacks”. ScienceDirect, Computers & Security
Ször P, Ferrie P. 2001. “Hunting for metamorphic". In: In Virus Bulletin Conference
E. Konstantinou. 2008. “Metamorphic Virus: Analysis and Detection,” RHUL-MA-2008-02, Technical Report of University of London. http://www.rhul.ac.uk/mathematics/techreports
Nguyen Anh Quynh and Kuniyasu Suzaki. 2010. “Next-generation debugger for malware analysis”. Black Hat USA
Microsoft msdn. Debugging functions
P. Ferrie. 2011. “The ultimate anti-debugging reference”.
Microsoft. 2018. Structured exception handling
Infosec Institute. 2015. “Zeroaccess malware - part 1”.
Manuel Egele, Theodoor Scholte, Engin Kirda, and Christopher Kruegel. 2012. “A survey on automated dynamic malware-analysis techniques and tools”. ACM computing surveys (CSUR), 44(2):6.
Md Shahnawaz, Bishwajit Prasad Gond and Durga Prasad Mohapatra. 2025. “Dynamic Malware Classification of Windows PE Files using CNNs and Greyscale Images Derived from Runtime API Call Argument Conversion”. Cornell University, arXiv:2505.24231v1
Ilker Kara. 2023. “Fileless malware threats: Recent advances, analysis approach through memory forensics and research challenges”. Expert Systems with Applications 214 (2023) 119133
Wueest, C., & Anand, H. 2017. “Internet security threat report-living off the land and fileless attack techniques”. An Istr Special Report. (pp. 4-9).
Bozkir, A. S., Tahillioglu, E., Aydos, M., & Kara, I. 2021. “Catch them alive: A malware detection approach through memory forensics, manifold learning and computer vision”. Computers & Security, 103, Article 102166. https://doi.org/10.1016/j.cose.2020.102166
Aljawarneh, S. 2011. “A web engineering security methodology for e-learning systems”. Network Security, 2011(3), 12–15. https://doi.org/10.1016/S1353-4858(11)70026-5
Menu