Mastering Malware Analysis

Mastering Malware Analysis, 2nd Edition · Alexey Kleymenov & Amr Thabet ·550 pages

Complete malware analysis methodology — from triage through static/dynamic/reverse engineering analysis. Covers anti-analysis bypass, APT techniques, MITRE ATT&CK mapping, IoT malware, and report writing.

Capabilities (10)
  • Apply triage workflow: hash → VirusTotal → strings → imports → entropy
  • Analyze PE file format: headers, sections, imports, entropy for packed/encrypted indicators
  • Map suspicious API imports to malware capabilities (injection, persistence, crypto, keylogger)
  • Conduct behavioral analysis: monitor process tree, files, registry, network in sandbox
  • Identify anti-analysis techniques: debugger checks, VM detection, packing/obfuscation
  • Reverse engineer XOR decryption loops and API hash resolution patterns
  • Map malware behaviors to MITRE ATT&CK tactics and techniques
  • Identify fileless malware indicators: LOLBins, PowerShell reflection, process hollowing
  • Write YARA detection rules from static and behavioral indicators
  • Structure analysis reports for threat intelligence vs. incident response audiences
How to use

Install this skill and Claude can walk through structured malware triage, interpret PE import tables and entropy indicators, explain anti-analysis bypass techniques, map observed behaviors to MITRE ATT&CK techniques, author YARA detection rules, and structure analysis reports for both incident response and threat intelligence audiences

Why it matters

Malware analysis is the foundation of threat intelligence and detection engineering — without it defenders react to symptoms rather than understanding the actual threat; a rigorous methodology that combines triage, static, dynamic, and reverse engineering phases ensures analysts extract maximum intelligence and don't miss sandbox-evading techniques

Example use cases
  • Running a five-minute triage on a suspicious PE binary — hash lookup, strings extraction, import table review, and entropy check — to determine whether full analysis is warranted
  • Authoring a YARA rule from a set of malware samples sharing a common loader by extracting distinguishing string and API call patterns without triggering on benign software
  • Mapping observed malware behaviors (encoded PowerShell, scheduled task persistence, LSASS memory access) to specific ATT&CK technique IDs to drive detection coverage gap analysis

Mastering Malware Analysis Skill

Analysis Workflow

Triage (What type? Safe to analyze?)

Static Analysis (Without executing)

Dynamic Analysis (Execute in sandbox)

Advanced Analysis (Reverse engineer if needed)

Report (For your specific audience)

Malware Categories

CategoryPrimary GoalKey Indicators
RansomwareEncrypt files, extortMass file rename, crypto API calls, ransom note
RATRemote controlPersistent C2, keylogging, screenshot capture
TrojanDeliver payloadMasquerades as legitimate file
RootkitPersistence + stealthKernel-level hooks, hide processes/files
BotnetDDoS, spam, click fraudBeaconing, IRC/HTTP C2, P2P
StealerCredential theftBrowser DB access, keylogger, clipboard
Dropper/LoaderStage 1 deliveryDownloads/decrypts next stage
WiperDestructionMBR overwrite, mass file deletion
FilelessMemory-onlyNo disk artifacts, PowerShell/WMI abuse

Static Analysis Techniques

Triage (First 5 minutes)

1. Hash the file (MD5, SHA256) → search VirusTotal
2. file command → PE? ELF? Script? Office doc?
3. strings → any plaintext URLs, registry keys, function names?
4. Import table → what Windows APIs does it use?

PE File Format Analysis

MZ header (0x4D5A)

DOS stub

PE header (NT headers: signature + FileHeader + OptionalHeader)

Section table (.text, .data, .rsrc, .rdata, ...)

Sections

Key fields to examine:

  • TimeDateStamp: compilation timestamp (can be faked)
  • Imports (IAT): DLL + function imports reveal capabilities
  • Exports: if any, suggests it’s a DLL or plugin
  • Resources (.rsrc): often contains embedded payloads
  • Entropy: high entropy (~8.0) in .text or .data → packed/encrypted
  • Section names: unusual names suggest custom packer

Suspicious Import Patterns

Imports SuggestCategory
VirtualAlloc + WriteProcessMemory + CreateRemoteThreadProcess injection
RegSetValue + HKEY_LOCAL_MACHINE\SOFTWARE\RunRegistry persistence
CryptEncrypt / CryptGenRandomEncryption (ransomware, C2)
WSASend / HttpSendRequestNetwork communication
GetKeyState / SetWindowsHookExKeylogger
OpenProcess + ReadProcessMemoryCredential theft (LSASS)
IsDebuggerPresentAnti-analysis

Dynamic Analysis (Behavioral)

Sandbox Setup

  • Isolated VM (no network except optionally INetSim/FakeNet-NG)
  • Snapshots before execution
  • Monitor: process creation, file system, registry, network

Monitoring Tools

  • Process Monitor (ProcMon): file, registry, process events
  • Process Hacker: process tree, memory regions, handles
  • Wireshark/FakeNet-NG: network traffic capture
  • Regshot: registry diff before/after execution
  • Cuckoo Sandbox: automated behavioral analysis

What to Look For During Execution

  1. Child processes: unexpected spawning (cmd.exe, powershell.exe)
  2. File drops: files created in %TEMP%, %APPDATA%, system directories
  3. Registry writes: Run/RunOnce keys, service creation
  4. Network connections: C2 domains, beaconing intervals
  5. Injection: handles opened to other processes, memory writes
  6. Defense evasion: AV/EDR process termination, log clearing

Anti-Analysis Techniques

Anti-Debugging

TechniqueHow It WorksDetection
IsDebuggerPresentReads PEB flagHook / patch the flag
CheckRemoteDebuggerPresentVia NtQueryInformationProcessMonitor API calls
Timing checksRDTSC delta too large = debuggerObserve behavior difference
Exception handlingDebugger catches exceptions differentlyStep through exception handlers
Heap flag checkPEB heap flags differ under debuggerManually check PEB

Anti-VM / Anti-Sandbox

  • Check for VM artifacts: VBOX, VMWARE in registry, VBoxMouse.sys driver
  • Check CPU core count (sandbox often has 1 core)
  • Check process list for sandbox agents
  • User interaction checks: mouse movement, clicks, uptime
  • CPUID checks for hypervisor bit

Bypass: use a “clean” VM with >2 cores, real user activity simulation, extended uptime.

Packing / Obfuscation

  • Packers: compress/encrypt code, decompress at runtime (UPX, custom)
  • Polymorphic: code mutates with each infection
  • Metamorphic: functionally equivalent but structurally different code

Bypass: let it unpack itself (run to OEP = Original Entry Point), dump process memory after unpacking.


Code Analysis (Reverse Engineering)

Tools

  • Ghidra (free) or IDA Pro: disassembly + pseudocode decompilation
  • x64dbg/OllyDbg: dynamic debugging
  • FLOSS: extract obfuscated strings from binaries

Key Patterns to Identify

; XOR decryption loop (common for string decryption)
mov ecx, [key]
xor [data], ecx
add data, 4
loop decrypt_loop

; API hash resolution (avoids import table)
call GetFunctionByHash
; then uses returned pointer — find hash→function mapping

C2 Protocol Identification

  • Look for: DNS queries with base64-encoded subdomains, HTTP requests with unusual headers/paths
  • Custom protocols often use XOR or RC4 with hardcoded keys
  • Search for: crypto constants (Rijndael S-box values, RC4 patterns)

APT Indicators and MITRE ATT&CK

MITRE ATT&CK Framework Usage

Map observed behaviors to MITRE tactics and techniques:

  • Tactic: what goal (Execution, Persistence, Lateral Movement…)
  • Technique: how (T1059.001 = PowerShell, T1053 = Scheduled Task…)
  • Sub-technique: specific variant

Common APT Persistence Mechanisms

  • Registry Run keys (T1547.001)
  • Scheduled tasks (T1053.005)
  • WMI event subscriptions (T1546.003)
  • Service creation (T1543.003)
  • DLL search order hijacking (T1574.001)

Fileless Malware Indicators

  • Living-off-the-land (LOLBin) abuse: mshta, regsvr32, wmic, certutil
  • PowerShell with -EncodedCommand or reflection-based loading
  • WMI for execution and persistence
  • Process hollowing / doppelganging
  • No files on disk → memory forensics required (Volatility)

Analysis Report Structure

For Threat Intelligence Audience

  1. Executive summary (business impact)
  2. Malware family / attribution (if known)
  3. Key indicators of compromise (IOCs): hashes, IPs, domains, registry keys
  4. MITRE ATT&CK techniques observed
  5. Recommendations

For Incident Response Audience

  1. Infection vector
  2. Execution chain (step-by-step)
  3. All IOCs with context
  4. Detection signatures (YARA rules)
  5. Remediation steps

YARA Rule Pattern

rule Ransomware_Generic {
    strings:
        $s1 = "Your files are encrypted" nocase
        $s2 = ".locked" nocase
        $api1 = "CryptEncrypt"
        $api2 = "FindFirstFile"
    condition:
        2 of ($s*) or all of ($api*)
}