4 open-source Mitre ATT&CK test tools compared

One way to learn how to better defend your enterprise is to train a red team to simulate attacks. The Mitre ATT&CK framework, which can be a very useful collection of threat tactics and techniques for such a team. The framework classifies and describes a wide range of attacks. To make it even more effective, various commercial and open-source general testing tools have been built to complement its schemas.

I examine four of the open-source tools: Endgame’s Red Team Automation (RTA), Mitre’s Caldera, Red Canary’s Atomic Red, and Uber’s Metta. Many others are either fee-based (such as Safebreach, AttackIQ’s FireDrill, or Verodin) or focus on limited use cases. All four are free and require a varying degree of supporting infrastructure. I tried them out on a test network of both Windows 7/10 and Mac endpoints to see how they work and what kinds of reports and insights they provide. 

Selecting and using ATT&CK testing tools

Before using any of these products, think about what you will be testing and how diverse your endpoint population will be. If you are primarily concerned with Windows, then all of them are appropriate. If you also want to examine the impact on Mac and Linux endpoints, you will need to look at Atomic Red or Metta.

CSO / IDG

Once you start your testing, I recommend first setting up at least one Windows virtual machine (VM) and disabling Windows Defender or other antivirus (AV) programs. Metta has a somewhat different testbed, but it also uses VMs. Turn off any AV screeners, because they could block some of the activities of the ATT&CK-based products that simulate threats.

Second, pay attention to which version of supporting software you will be using. Some of the products (such as RTA) only work on specific Python versions. Finally, for the three Python-based tools, find someone with existing Python skills. While you don’t need a lot of Python knowledge to use these tools, this shouldn’t be your first Python project.  

Each tool has a varying degree of prerequisites to get it up and running, from the bare minimum for Atomic Red to a rather complex collection for Caldera and Metta. Some of the tools have better documentation (like Caldera and Atomic Red) and some will require more careful study (like Metta). The good news is that the developers of all these tools will take your suggestions to improve their documentation.

You can use these tools to satisfy several different security needs. First, any of them are useful to learn more about the Mitre ATT&CK tactics and illustrate how particular malware and methods can penetrate your network. If you aren’t sure your defensive measures are comprehensive and complete, using any one of these tools will quickly tell you where you need to beef things up, change configurations, close firewall ports, and so forth.

Second, each tool focuses on a different collection of the Mitre tactics, and none cover all the various conditions laid out in the different framework ATT&CK matrices. Comparing the raw number of tactics isn’t relevant, because this actual number doesn’t really represent what conditions you may or may not be looking for.

This is probably the most time-consuming part of using these tools: understanding what circumstances they cover and what they don’t cover. Part of the challenge is that all four tools — like the Mitre ATT&CK framework itself — are under active development, and new features are regularly added.

My suggestion is to start with the Atomic Red tool if you are new to ATT&CK or to red team testing. Spend some time learning more about the Mitre matrices and methods, then move to Endgame’s RTA once you learn your way around the ATT&CK framework and want to investigate specific tactics.

That might be sufficient as a stopping point, but if you are part of a larger red team then you might want to stand up the infrastructure for Metta or Caldera for further investigations. If you are frustrated by the lack of reports and analysis support, you may want to move on to the fee-based tools because they offer much more in these departments.

All four products can produce a security playbook of automated routines but do them differently, from the bare-bones style that Atomic Red uses to the more sophisticated scenarios that Metta can handle. What level you will ultimately need is more a matter of taste and comprehensiveness and your circumstances.   

Red Canary Atomic Red

The most bare-bones of the four tools is Atomic Red, which has nothing in the way of software to download and configure. It differs from the other tools in that it isn’t based on Python, or any other scripting language. Instead, it is composed of separate instructions (rather than scripts per se) that match specific ATT&CK tactics.

You start with going to a webpage with its version of the ATT&CK matrix for your particular OS. The cells for which Atomic Red has built attack scenarios are hyperlinked to their descriptions in their Git repository. Like the Mitre matrices, there are separate ones for Windows, Mac and Linux endpoints. About half the cells in the matrix have links to these instruction pages.

The reason why I say the download requirement is nil is a matter of interpretation. There are no executables or scripts in the entire tool. Instead, you get more of an instruction manual to navigate the different ATT&CK scenarios. Yes, you will need some software, but the difference is that Atomic Red leverages existing hacking tools that you probably already have installed.

Let me explain using a particular testing scenario, say the Windows Remote Management technique that Mitre has identified as T1028. In this case, you make use of Mimikatz and PowerShell using the MMC20 Windows application to move laterally across the network. The documentation under this tactic gives you a sample command line sequence to execute the exploit and a hyperlinked webpage that goes into more details (in this case, a page on CobaltStrike’s blog).

For each specific ATT&CK tactic, you need to use a particular attack and observe the results. Think of Atomic Red as more of a teaching tool for how to perform particular attacks using the command syntax of particular OS functions. This is probably is something you once learned but have since forgotten. If you aren’t big on long strings of command line parameters, this can be a useful reference or a crib sheet for you to cut and paste the parameters into your own endpoint to see what happens.

Atomic Red also has a special “chain reaction” feature, where you can combine different tactics into a single overall sequence to better simulate complex attacks or to create your own security playbooks. One example of this is a series of Visual Basic code that Red Canary calls Dragon’s Tail.

Even this is incomplete; you’ll need to find the particular malware-embedded Word documents on your own that launch this kind of attack. It is more meta-malware than the actual malware itself. The Atomic Red chain reaction shows you how some malware operate, without giving you the actual thing itself. I found this less than satisfying and somewhat frustrating.

One downside to Atomic Red is that unlike the other tools, it generates no reports to summarize what you have learned from these exercises. Of course, you can purchase the managed service that Red Canary sells that has these features, but if you want to stick to what is available for free, you have to collect evidence of the exploit on your own or use some other endpoint detection and response (EDR) analysis tool. Atomic Red developers designed it this way to have as low a barrier to entry as possible for its use.

Red Canary

Endgame RTA

The next step up in terms of ease of use is Endgame’s RTA. It was designed to be a simple install with minimum requirements. It contains nearly 50 different attacks that are packaged in their own separate Python scripts.

Endgame

The tool originated out of a need by the Endgame developers who were looking for a better way to test their own endpoint detection product and use more automated quality assurance methods. “We needed a unit test that would be able to emulate the bad behaviors we were trying to detect with our product line,” said an Endgame developer. When they finished building RTA, they realized that it could be useful for IT security analyst and released it as open source.

Installing RTA isn’t really an issue. You just download the scripts and run them on whatever Python installation you have. You will need the 2.7 Python version because it doesn’t yet support the 3.x version. RTA does have a few prerequisites with the SysInternals tools, but that is easily downloaded. It works with only Windows endpoints, although they plan to add Mac and Linux in future releases.

Unlike Atomic Red, you get no handy hyperlinked reference guide that maps the scripts to the individual ATT&CK tactics. You will have to know in advance which ones you are interested in and suss out the names of the scripts that are part of the toolset. The actual Mitre ATT&CK tactic designation (like T1107) is documented in the Python code itself so you can use that and go to the ATT&CK wiki to find the right match.

An improvement over Atomic Red is actual output that you can examine once the script is run that illustrates the particular exploit. Some of the scripts take several minutes to gather their data, so be patient. Given that you are dealing with what can be seen on the command line, that output might not make any sense to you and send you back to the ATT&CK framework website for further interpretation.

To create a playbook, you can batch together the individual scripts with a simple Python IF statement for more hands-off automated operations. This isn’t all that sophisticated, so if you want something more complete, you will probably end up using either Caldera or Metta.   

Mitre Caldera

The Mitre team that developed the ATT&CK framework also has developed its own red team tool called Caldera. Unlike Atomic Red, it has a rather lengthy installation, although very well documented. Caldera has three component pieces: a server, an agent, and a separate Windows program called Crater that is used for adversary emulation exercises.

You can install agents (using admin rights) on any 64-bit Windows endpoint starting with Windows 7. The server runs on either Windows or Linux and requires a full Windows domain and Python v3.5.4, (ensuring that you upgrade its setuptools component to v24), and MongoDB v3. If installing on a Windows server, you’ll also need Microsoft’s Visual C++ tools. All these requirements and supplementary installation and configuration steps are well documented online.

Once you have installed all the prerequisite software, you will need to create your test enterprise Windows network to appear more like a “real” Windows network. This involves a few different activities. Again, they are well documented but still somewhat tedious. First you must introduce something called credential overlap. This is where a domain user has admin rights on two Windows 10 computers in your test domain. It is used to illustrate a few different attack scenarios.  There are other adjustments for Windows 8 or 10 Registry values, too. The proces will take about an hour or longer, depending on how carefully you read the instructions and whether you understand them.

Caldera has adversaries, networks and operations. The former represents a real adversary’s tactics that are created in the web server console. An adversary contains several steps that match the Mitre matrix cell designations. The network is what the adversary operates on and are just collections of targeted hosts. The operation pulls these two elements together, and they contain the starting IP address of the host that will run a remote Trojan automatically. Once you create the operation, it immediately runs.

Mitre

Caldera displays the results of its operations in a status window on the web console (see the screenshot below) and from reading its own log files. These are designed for machine rather than human consumption, similar to the other tools reviewed here. 

Mitre

Caldera has a lot more prerequisites than RTA and for not much more added benefit. It does preserve the state of an attack’s origins so you can better troubleshoot your own network’s flaws and lack of particular defenses. The developers claim this can better show defenders how adversaries actually operate and try to penetrate your networks.Mitre

Uber Metta

Metta is another complex install and sadly not as well documented as Caldera. It depends on several prerequisites but doesn’t use an agent/server configuration. Everything is installed on one Windows or Linux or Mac endpoint that runs a series of VMs and command lines for the tests. You can mix and match the OS that you run the Metta code on and what the OS target of your testing will be.

What confused me initially is that the installation sets up these VMs using a series of automated processes playing off of Redis, Celery, Vagrant, Virtual Box and a few other odds and ends. I set everything up on my Mac, but you could have just as easily set up the tool on either Windows or Linux.

Metta has some installation instructions, and for the most part if you follow along you can get things working in about an hour. I stumbled over some of the configuration files, which you must edit manually depending on what file locations you installed all the software mentioned above. Once you get everything running, you then use a series of Python scripts that control its operation, similar to the other tools. Like RTA, each script is matched to a particular Mitre ATT&CK tactic number. Unlike RTA, it contains the hyperlink to the ATT&CK wiki to easily look up what it is supposed to be testing—a nice touch.  

Uber

Once you get at least one test VM created – in my case I used the sample Vagrant StefanScherer Windows 10 image that you can download online – you just let it run inside Virtual Box and ignore it, as Metta orchestrates its simulated attacks on that endpoint. Metta is designed for you to bring up two command lines: one running the actual Metta files (which are just Python scripts) and the other more of a debug display from the Celery process that will show you the impact of those script commands. It takes a bit to get used to it, and what Metta is designed to do is to create a bunch of log files that you then parse either with an EDR tool or some other automated method.

Metta works with a series of actions and scenarios. Actions are particular tests such as issuing a command to query the network status of your endpoint. Scenarios are composed of a list of paths to particular sets of actions.

Metta has much more output to examine than some of the other tools reviewed here, but not in very human-readable form. If creating security playbooks is your main goal, then Metta is probably the best product for this purpose. If you are a single-person red team, this is probably more work than you’ll want to deal with.