CTI-Agent: Automated battlecards from CTI reports

Threat intelligence reports from reputed parties contain a wealth of OSINT including threat actor details, campaign information, IOCs (indicators of compromise), and TTPs (Tactics, Techniques and Procedures). Such threat intelligence is predominantly consumed with a human in the loop due to several challenges posed: Threat intelligence is often in natural language and difficult to extract automatically; These reports may have incomplete information and may require synthesizing multiple reports to construct a better view of the attack; Some intelligence such as TTPs are often implicit in the report and requires language comprehension; Not all indicators in a report are malicious and further they could have different degrees of confidence on the level of maliciousness and what they define as malicious.

The labor intensive manual process not only makes it difficult/error prone to identify actionable threat intelligence in the form of battlecards but also leave users vulnerable to mentioned attacks due to the increased time gap threat reports and manual extraction of intelligence. The problem is exacerbated by the fact that many similar threat reports with different pieces of intelligence scattered across reports especially for emerging attacks.

We build an agentic system to automate the collection and synthesis of cyber threat intelligence from threat reports using LLM Agents and unsupervised machine learning techniques into battlecards. At a high-level, CTI-Agent first extracts threat actor, campaign, TTPs and IOCs from recently published threat reports from reputed parties using specially crafted prompts on LLMs (Large Language Models) as well as using regular expressions/known knowledge which we refer to as signature based techniques. The agent also generates concise summaries for each threat report using LLMs. After performing a round of validation, the agent uses the summaries and extracted intelligence to synthesize multiple reports together and provide a battlecard with easily digestible threat intelligence. The agent follows the proven ReAct (Reason Action) framework to plan tasks autonomously and achieve the final goal of producing accurate battlecards by reasoning and then acting (i.e. calling various tools) multiple times. We plan to share our experience and lessons learnt during the process of build the CTI-Agent.

The outline of the presentation is as follows:

CTI to Battlecards

How battlecards are used to help protect networks

Manual, time consuming, error-prone

Multiple threat reports with inconsistent descriptions

May contain conflicting IOCs/TTPs

Modeling CTI Reports

Converting unstructured or semi-structured data into structured threat information

Challenges involved

Three key LLM patterns

Prompting LLMs (simple and CoT prompting)

RAG (Retrieval Augmented Generation)

Agents

Prompting LLMs

How to effectively prompt LLMs to elicit best output

Examples

RAG

Describe a RAG system using a diagram

Agents

Describe an magnetic system using a diagram

Evals

Evaluating LLM/Agentic systems is a challenging task

Show how one can incrementally build an eval dataset to evaluate

Agent Tool Calling

Introduce Agent tool calling

Introduce MCP protocol

Multi-Agent Systems

Common patterns

Introduce A2A protocol

Popular Agent Planning Techniques

Introduce what agent planning is

Introduce patterns like Reflection and ReAct

Guardrails

Explain the need to have guardrails

Provide examples

Multi-Agentic System Overview

Monitor and collect recent threat reports from reputed parties

Agentic System to extract Threat Actor, Campaign, TTPs and IOCs

Extract using CoT prompted LLMs

Extract using signature based methods

Validate the collected threat intelligence information via reflection and LLM-as-a-Judge

Create threat report summaries for each threat report prompting LLMs

Collect additional IOCs related to campaigns using in-house intelligence

Save reports, summaries, threat intelligence data to a database

Cluster threat reports to identify related threat reports (i.e. those reports discussing the same threat or campaign)

Generate language embeddings for the threat summaries for threat reports

Generate graph embeddings by modeling threat reports and threat intelligence extracted as a graph and using unsupervised graph learning algorithm

Combine both embeddings together and perform unsupervised learning to cluster embeddings together

The embeddings in the same cluster correspond to threat reports discussing the same threat or campaign

Generate battlecards that can be readily used by security operations professionals

Note: The above steps will be visualized into multiple slides and showed how to realize them in practice.

Agentic System Evaluation

Dataset

Experimental results

Lessons Learned

Various lessons learned during the construction and evaluation of this system plus several other agentic systems that the author built

Summary

Key take aways from the presentation