JF / 2026
Available
Tool · Malware Analysis · 2026.07.08 · Intermediate · 24 min read

CAPE Sandbox — Analyzing Malware Without Becoming the Next Victim

A practical workflow for running untrusted binaries in a controlled environment: how CAPE extracts behavior and IOCs, why anti-VM evasion breaks most sandboxes, and the configuration changes that make a VM look like a real machine.

Watch video ↗

The premise

There are two ways to analyze a piece of malware. Static analysis examines the file without executing it — reading the code, examining the structure, extracting strings and embedded resources. Dynamic analysis executes the file in a controlled environment and observes what it does. Each approach reveals different things, and the combination is more powerful than either alone.

The problem with dynamic analysis is that executing malware is dangerous. A binary that encrypts files, steals credentials, or opens a remote backdoor will do those things on whatever machine it runs on. The analyst needs an environment where the malware can execute freely — so that its behavior is visible — while the rest of the world is insulated from the consequences.

The standard solution is a virtual machine with a malware analysis sandbox. The VM isolates the malware from the host, and the sandbox automates the observation: it monitors the filesystem, the registry, the network, the process tree, and the API calls, then produces a report with the findings.

CAPE is one of the more capable open-source sandboxes. It is a fork of Cuckoo Sandbox, maintained by an active community, with significant enhancements over the original — particularly in payload extraction (the “Config And Payload Extraction” that gives it its name) and anti-evasion detection. It is the sandbox of choice for researchers who need more than what a simple automated tool provides.

This writeup covers what CAPE does, how to set it up so that it actually observes the malware instead of being evaded, what the output tells you, and the safety practices that prevent a research session from becoming an incident.

Part one — what CAPE is

CAPE began as a fork of Cuckoo Sandbox, which was the standard open-source automated malware analysis tool from around 2010 until its development slowed in the late 2010s. CAPE picked up where Cuckoo left off and added capabilities that the original lacked.

Behavioral monitoring. Like Cuckoo, CAPE runs the sample in a VM and instruments the operating system to log everything the malware does: file operations, registry modifications, process creation, network connections, API calls, and window titles. The instrumentation is at the API level, which means the sandbox sees the malware’s intent, not just its effects.

Payload extraction. The feature that gives CAPE its name. Many malware samples are packers or loaders — they contain an encrypted or obfuscated payload that is decrypted at runtime and executed. A traditional sandbox sees the outer binary’s behavior but misses the inner payload. CAPE monitors for the moment of decryption and extracts the payload from memory, so the analyst can analyze the actual malware rather than the wrapper.

Anti-evasion detection. CAPE includes specific monitors for the techniques that malware uses to detect that it is running in a sandbox: checking for VM artifacts, timing checks, user interaction checks, and human activity simulation. When the sandbox observes these checks, it can flag them, and the analyst can adjust the VM configuration to pass them.

Configuration extraction. Many malware families embed their C2 configuration — the domains, IPs, ports, and encryption keys that the malware uses to communicate with its operator. CAPE includes parsers for dozens of families that extract this configuration from the sample and from memory, producing structured output that can be fed into threat intelligence platforms.

Flexible reporting. The output is a JSON report with the full behavioral trace, plus a web interface for browsing the results. The JSON can be processed with jq or imported into other tools. The web interface provides timeline views, process trees, network graphs, and searchable logs.

The combination — behavioral monitoring plus payload extraction plus anti-evasion detection plus configuration extraction — is what distinguishes CAPE from simpler sandboxes. The installation is more involved, and the configuration matters more, but the quality of the analysis is significantly higher.

Part two — the environment

The setup is where most analyses succeed or fail. A poorly configured sandbox produces garbage, and most public writeups on “malware analysis” skip this part entirely.

The hypervisor. CAPE runs the VM in VirtualBox, VMware, or KVM. KVM is the recommended choice for production setups because of its performance and its ability to be automated. For a single-analyst setup, VirtualBox is the simplest — CAPE has built-in support for it, and the configuration is less involved.

The guest OS. The sample determines the guest. Most Windows malware expects a modern Windows environment — Windows 10 or 11 is the safe default. Some older families expect Windows 7 or even XP, and some run differently on different versions. If the analyst does not know what the sample targets, running it on multiple guests reveals more.

The guest configuration. This is the part that most tutorials skip. The VM must look like a real machine, not a fresh installation of Windows. A minimal Windows VM has:

This is the profile of a sandbox, and malware authors know it. Samples that check for these characteristics will refuse to execute, exit silently, or present benign behavior — and the analyst will conclude that the sample does nothing.

The fix is to make the VM look used:

The principle is simple: the more the VM looks like a real machine, the more of the malware’s behavior you will observe. The details of what counts as “real” change over time, as malware authors update their detection techniques, but the general strategy — populate the VM, customize it, make it look used — is durable.

Part three — networking

The network configuration is the second most important decision, and it determines both the safety and the quality of the analysis.

Completely offline. The VM has no network adapter. This is the safest configuration — the malware cannot communicate with its C2, so it cannot receive commands or exfiltrate data. The downside is that the analyst sees only the malware’s initial behavior, not its full capabilities. For some samples, this is enough. For others, the interesting behavior happens after the C2 connection.

Isolated network with simulated services. The VM is connected to a private network where CAPE runs services that simulate the internet: DNS, HTTP, HTTPS, SMTP, and various other protocols. The malware connects to what it thinks is its C2, and CAPE logs the connection and can respond with fake data. This is the standard configuration for automated sandboxes — it provides some of the behavioral richness of a live connection without exposing the analyst to real risk.

CAPE includes an “inetsim” component that provides these simulated services. It responds to DNS queries with the sandbox’s IP, it answers HTTP requests with default responses, and it logs everything. The malware’s C2 traffic is captured, but it never reaches a real server.

Live internet with a VPN. The VM has internet access through a VPN. This is the most dangerous configuration — the malware can communicate with its real C2, download additional payloads, and exfiltrate data. It is also the most informative, because it reveals the full attack chain.

The risk is not just that the malware does damage. It is that the malware’s C2 infrastructure can identify the analyst. If the sample has a “kill switch” or a fingerprinting mechanism, connecting to it from a live IP may alert the operators that they are being analyzed. Some families have been known to delete their infrastructure when a sandbox connects.

The recommendation: use the isolated network with simulated services for the default case. Move to live internet only when the analysis requires it, and only behind a VPN that is dedicated to the analysis (not shared with personal use).

Part four — executing the sample

Once the VM is configured and the network is set, the actual analysis begins.

The submission process. The sample is submitted to CAPE through the web interface or the API. CAPE copies the sample into the VM, executes it, and monitors the behavior for a configured duration (typically two to five minutes, adjustable).

During execution. The sandbox logs everything. The analyst can watch the process tree in real time through the web interface, seeing which processes spawn, which files are touched, which network connections are made. This real-time view is useful for understanding the malware’s progression and for spotting behavior that the automated report might not emphasize.

Post-execution. After the timeout, CAPE produces a report. The report includes:

Part five — reading the report

The report is large, and the analyst needs to know what to look at first.

The process tree. This is the starting point. It shows how the malware executed — did it spawn a child process, inject into another process, use rundll32 or regsvr32 as a proxy, or run entirely within its own process. The tree reveals the malware’s execution strategy.

The network connections. This is often the most valuable section. The domains and IPs the malware contacted are IOCs that can be used to block the threat elsewhere. The connections also reveal the malware’s purpose — connecting to a known C2 domain, a cryptocurrency mining pool, a pastebin URL, or an advertising network.

The dropped files. Any files the malware wrote to disk are candidates for further analysis. A file dropper that writes a DLL and executes it via rundll32 is a two-stage attack, and the DLL is the second stage.

The extracted payloads. If CAPE captured a decrypted payload, this is the actual malware — the code that does the work. Analyzing the payload reveals the full capability set.

The signatures. CAPE’s signature database matches the sample against known families and behaviors. A signature match to Mirai, Emotet, or Cobalt Strike tells the analyst what family the sample belongs to and what to expect. False positives are possible, so the signature is a starting point, not a conclusion.

The configuration. If the sample is a known family with a parser, CAPE extracts the C2 configuration. This is often the most actionable output — the domains, IPs, and encryption keys that the malware uses, which can be fed into defensive systems.

Part six — anti-evasion in practice

The point of configuring the VM to look real is to defeat the malware’s detection of the sandbox. This section covers the specific checks that malware performs and how to pass them.

VM artifacts. Registry keys (HKLM\SOFTWARE\VMware, Inc., HKLM\SOFTWARE\Oracle\VirtualBox), specific files (C:\Windows\System32\drivers\vmmouse.sys), and specific device names are checked. Removing these artifacts is difficult — they are part of the hypervisor’s presence — but some can be hidden through registry cleanup and driver removal. The VBoxGuestAdditions and VMware Tools installations are the most visible, and uninstalling them removes some of the artifacts.

MAC address prefixes. VirtualBox and VMware assign MAC addresses with specific OUI prefixes (08:00:27 for VirtualBox, 00:0C:29 and 00:50:56 for VMware). Malware can check the MAC prefix to detect the hypervisor. The fix is to change the MAC address to a prefix from a real hardware vendor — 00:1A:2B (Ayecom), 00:25:00 (Apple), or similar.

CPU count and RAM. VMs often have fewer CPU cores and less RAM than real machines. A VM with 1 vCPU and 2 GB RAM is a clear sandbox signature. Configuring the VM with 4 vCPUs and 8 GB RAM matches the profile of a typical modern workstation.

Disk size. A 40 GB disk is the VirtualBox default and is a VM tell. Increasing to 120 GB matches a real machine.

User interaction checks. Malware that wants to avoid sandboxes may check for user activity: mouse movement, window focus, keyboard input. If the sandbox is fully automated and no user interaction occurs, the malware may conclude it is in a sandbox. The fix is to simulate user activity — either with a script that moves the mouse and types periodically, or by manually interacting with the VM during the analysis.

Timing checks. Malware can measure how long a system call takes and detect the instrumentation overhead of the sandbox. This is harder to defeat — the overhead is real — but it can be reduced by optimizing the sandbox configuration and using a faster hypervisor.

Sleep and delay. Many samples use Sleep or SleepEx to delay their execution, waiting for the sandbox’s timeout to expire before doing anything interesting. CAPE includes a “sleep skip” feature that intercepts sleep calls and advances the clock, but the malware can detect this. The fix varies by sample; sometimes the analysis just needs to run longer.

The general principle: configure the VM to look like a real machine, and address the specific detection techniques that the sample uses. There is no universal configuration that defeats every anti-analysis technique. The analyst adapts the environment to the sample, not the other way around.

Part seven — the safety practices

The purpose of the sandbox is to prevent the malware from affecting anything outside the VM. The practices that ensure this are non-negotiable.

Isolate the sandbox host. The host machine should not be used for anything else. No personal browsing, no email, no work files. The host is a dedicated analysis machine, and the only network access it has is what the sandbox requires.

Do not use shared folders with the host. VirtualBox and VMware allow shared folders between the host and the guest. These are a bidirectional channel — malware in the guest can write to the host’s shared folder. Disable shared folders for the analysis VM, or use a one-way mechanism (like a read-only shared folder) if the analyst needs to transfer files.

Do not enable clipboard sharing. The clipboard is a similar channel. Malware can read and write the clipboard, which means it can potentially exfiltrate data from the guest to the host or influence the host’s behavior.

Do not mount USB devices from the host. USB passthrough connects the guest directly to the host’s USB subsystem. A malicious sample with USB control can attack the host through this channel. Keep USB devices disconnected from the analysis VM.

Isolate the network. The VM should be on an isolated network segment, not on the same network as the host’s other devices. This prevents the malware from scanning or attacking other machines. The sandbox’s simulated network services should run on this isolated segment.

Snapshot before every analysis. Take a snapshot of the clean VM state before executing a sample. After the analysis, revert to the snapshot. This ensures that residue from one sample does not contaminate the analysis of the next.

Never execute the sample on the host. This is obvious, but it happens. A sample that “looks like a text file” and turns out to be a Windows executable can be accidentally double-clicked. The sandbox is the only environment where the sample should be executed.

Treat the sandbox output as untrusted. The report contains IOCs — domains, IPs, hashes — that are potentially hostile. If the analyst imports them into a threat intelligence platform, the platform should treat them as indicators, not as trusted data. A domain that the malware contacted may be a legitimate service that the malware abused.

Part eight — the broader lesson

CAPE is a tool, and like any tool, its value depends on how it is used. A sandbox that is configured badly produces reports that miss the interesting behavior. A sandbox that is configured well produces reports that reveal the malware’s full capability.

The configuration that matters is not the sandbox software itself — it is the VM. The VM must look like a real machine to the malware. This requires effort: installing software, creating user data, customizing the environment, matching the profile of a typical workstation. The effort is what distinguishes an analysis that works from an analysis that produces garbage.

The same principle applies beyond malware analysis. Any environment that is designed to observe adversarial behavior — a honeypot, a deception platform, a threat intelligence sensor — must be indistinguishable from a real target. If the adversary can tell the difference, the observation fails. The design of the environment is the design of the observation.

For the analyst, the practical lesson is that the interesting behavior of malware is hidden behind layers of anti-analysis. The malware does not want to be seen. The analyst’s job is to see it anyway, by making the environment convincing enough that the malware forgets to check. The work of making the environment convincing is the work of malware analysis.

Takeaway

CAPE Sandbox is a capable open-source tool for dynamic malware analysis. It monitors behavior, extracts payloads, detects anti-evasion techniques, and produces reports that reveal what the sample does, where it connects, and what it tries to hide.

The tool is only as good as the environment it runs in. A default VM configuration produces analyses that miss most of the interesting behavior, because the malware detects the sandbox and refuses to execute. A configured VM — one that looks like a real machine, with installed software, user data, and a plausible hardware profile — produces analyses that reveal the malware’s actual capabilities.

The safety practices are non-negotiable. The sandbox is only safe if the host is isolated, the network is contained, the snapshots are taken, and the sample never touches anything outside the VM. A malware analysis session that breaks these rules is not an analysis — it is an incident.

The workflow is not glamorous. It is downloading a sample, configuring a VM, waiting for the analysis to complete, and reading a report. The value is in the details: the configuration, the anti-evasion work, the interpretation of the output. The analysts who do the details well are the ones who produce useful intelligence. The analysts who skip the details produce reports that no one uses.

← Back