Best practices to evaluate LLM-based AI security tools

Require an open-door policy

First and foremost, those adopting an LLM-type AI in cybersecurity should demand that AI not be a black box. It should be what is known as explainable AI, which enables response teams, both during the response and in after-action reviews, to see exactly what triggered the AI and incited a response. It also enables teams to understand what was missed and why an AI-based tool failed to respond to something that was later identified as an attack.

CISOs should also ask about the provenance of the AI model, including its base model, how it was trained and who implemented the training. Model maintenance is also important, so evaluators should ask how the model is retrained in the face of drift, as well as how it will be updated with new types of training data.

Evaluate AI functionality

The main reason AI-powered tools are attractive is their ability to bring real time responsiveness to traditionally human-only areas of cybersecurity. For example, tools can review alerts from defensive tools, such as firewalls and SIEM systems, to determine which require a response, or detect signals of a single compromise spread across alerts from multiple security systems.

CISOs must decide whether the AI functionality in the tool they're evaluating is a glorified help function or can actually solve these problems and assist with other human-intensive activities, such as writing scripts to drive incident responses.

Determine data sharing risk tolerance

AI-powered tools are typically trained on data from sites where they have been deployed. CISOs should understand their enterprise's risk tolerance for sharing internal operating data in this way and ensure evaluators are clear on it.

Make no mistake, there is risk. LLMs are fundamentally incapable of being leak-proof, so it is possible to get a model to disgorge any information it has been fed. Evaluation teams should ask whether their organization's data will be secured and used to train models for use elsewhere. They must understand how data will be redacted or anonymized before aggregation, and what layers of security outside the model will be used to prevent leakage.

That brings up the issue of data governance. Teams should ask where the organization's data will be stored -- especially if data sovereignty laws apply -- and what level of control the organization has. Teams should also ask if they can provide and control encryption keys for additional security.

Calculate ROI and costs

Cybersecurity tools don't generate revenue, but their absence can lead to revenue loss. ROI calculations for these tools generally focus on losses avoided and staff time saved. In evaluating options, teams should look for tools that can drive radical reductions in the mean time to contain threats. MTTC has three components, each of which AI can reduce: mean time to detect an attack, mean time to understand the nature of the attack and mean time to respond to contain it.

Reducing the amount of staff time required for each phase forms one component of ROI. Additional ROI components include reducing actual downtime and associated revenue losses and reducing the risk of regulatory fines, recovery and service restoration costs.

CISOs must balance ROI factors against the full cost of operating the AI-powered system, including any variable, per-token or per-API-call cost structure. The true magnitude of token-based costs can only be understood in the context of how the cybersecurity team will use the tool in that specific environment. Some vendors might charge additional fees for ingesting the enterprise's historical log data, which must be weighed against the benefits of using it to fine-tune the tool's model.

Although most vendors claim that such costs are essentially zero, CISOs must consider the time required to train staff on new systems. Beyond training, staff will also need to spend time updating security processes and policies to incorporate the new tool, as well as conducting additional war-gaming exercises to test staff and policies.

Who watches the watchmen?

CISOs must consider how the organization will monitor new LLM-based tools. The most basic backstop is to require thorough, auditable logging that goes into a system the tool cannot tamper with. Implement additional tools to regularly review logs to identify signs of drift, failures to detect obvious red-flag events and even hallucinations -- another inevitability in all LLMs.

The other mainstay backstop is to require human-in-the-loop process design wherever an agentic system is given power over critical services. Requiring a human to approve something -- such as quarantining the CIO's laptop or a primary e-commerce Kubernetes cluster -- might be the only thing standing between the cybersecurity team and serious political or financial consequences from a machine overreaction.

Whenever a new category of functionality emerges, CISOs and other security professionals need to revise their approach to product selection in light of new potential benefits and new potential risks. They are more likely to achieve a good result -- or at least avoid a bad one -- if they take new risks and rewards into account at every stage of evaluation.

John Burke is CTO and a research analyst at Nemertes Research. Burke joined Nemertes in 2005 with nearly two decades of technology experience. He has worked at all levels of IT, including as an end-user support specialist, programmer, system administrator, database specialist, network administrator, network architect and systems architect.