LLM Sanctions Screening Accuracy Put to the Test by Castellum.AI

Castellum.AI, a sanctions and watchlist data vendor, reported that nine standalone large language models (LLMs) cleared or escalated 30.8% to 71.1% of 500 screening cases correctly, against 98.1% to 99.6% when the same models ran inside its own software.

The company published the results in September as FCA Bench, short for FinCrime Agent Benchmark. The test puts LLM sanctions screening accuracy side by side with decision consistency, meaning whether a model reaches the same answer when it is run again on the same case.

For compliance teams weighing AI agents for alert review, the figures bear on a practical question: how much of the work an LLM can take on its own, and how much depends on the data and matching tools around it. Castellum.AI’s results are vendor-run and have not been validated by a third party, so they are best read as a test design to compare against a firm’s own pilots.

How the LLM sanctions screening accuracy test worked

Each model assessed 500 cases, according to Castellum.AI’s benchmark page. The set held 160 sanctions cases, 180 politically exposed person (PEP) cases and 160 adverse media cases, with exact matches, fuzzy matches and no-match names. It contained 152 true positives.

Every model completed three standalone runs and three runs inside the Castellum.AI Harness. Both configurations used the same cases and the same baseline standard operating procedure (SOP), the written rules that tell an analyst when to clear or escalate an alert.

The nine models are Claude Sonnet 5, Claude Fable 5.1, Gemma 4-31B, GPT 5.5, GPT 6 Astra, Grok 4.20, Kimi K2.5, Muse Spark 1.2 and Gemini 3.6 Flash. Muse Spark scored lowest standalone at 30.8%. GPT 6 Astra scored highest at 71.1%.

Standalone models received a fixed prompt that included the SOP, directions to the online risk sources Castellum.AI uses, and web search. They retrieved the information themselves, then decided. Castellum.AI graded each case on whether the model raised the right alert and adjudicated it correctly, producing one end-to-end pass or fail score.

Consistency gaps in standalone models

Average decision consistency rose from 45.2% for standalone models to 97.2% inside the harness. Standalone models gave different wrong answers across successive runs, Castellum.AI said.

Retrieval was a weak point. Standalone models found an average of 61% of expected hits, even with directions to the screening sources, Chief Executive Peter Piatetsky said. Some sanctions checks took minutes and cost more than $1 per screen as models located, analyzed and cleaned data.

The harness combines Castellum.AI’s risk database and screening tools with what it calls Arbiter agents, which apply an institution’s policies to sanctions, PEP and adverse media alerts. Arbiter supplies structured risk records and matching tools, so the LLM only assesses the alerts that result.

Piatetsky said judgment and adjudication suit LLMs better than primary screening. He also pointed to stronger results from open-weight models such as Gemma 4-31B inside the harness, which firms could consider for in-house deployments.

What the benchmark leaves unanswered

The public page does not say how the case labels were set. It states only that cases were built against Castellum.AI’s risk database. The company makes aggregate results and detailed methodology available to approved parties on request, which means outsiders cannot reproduce the run from public material.

The test used one baseline SOP and 500 cases, and the results are current as of September 16, 2026. Models change often, so a score for one version may not hold for the next.

The page also carries figures that do not reconcile with each other. It says harness consistency was 97.2% or higher across every model, while the stated range starts at 95.1%. A firm reading the benchmark would need the underlying data to resolve such gaps.

Tuning to each institution’s policy

Piatetsky said risk appetite differs by firm. One bank may welcome a high-risk customer whose business it understands, while another would refuse the same customer, so the agent has to apply each institution’s own procedures.

For client deployments, Castellum.AI asks for policies and procedures, a set of alerts with completed adjudications and a second set without. It tunes the agent against the completed cases using the client’s answer key, then tests on the unseen set. “And then we run it on basically the closed book test,” Piatetsky said.

Piatetsky said the process had found analyst mistakes in every deployment. Spencer Vuksic, vice president of growth, said testing can also expose faults in the written rules. “We also find inaccuracies or inconsistencies in the SOP itself,” he said, because policies specify which differences in names, locations, dates of birth and identifiers justify clearing or escalating an alert, and an agent can surface contradictions.

Piatetsky said the research responded to buyers who struggle to tell competing AI claims apart. He recalled a bank chief executive telling him: “I don’t know how to tell you guys apart.” He also noted that banks building their own agents must staff them after launch, asking who will maintain the engineering and manage the integrations.

Where regulators stand on LLM screening

No rule singles out LLMs for sanctions screening in the sources reviewed here. Existing frameworks on models and sanctions programs apply.

In the UK, the Prudential Regulation Authority’s Supervisory Statement 1/23 sets model risk management principles for banks. They took effect on May 17, 2024, apply to UK banks, building societies and designated investment firms with internal model approval, and cover models used to inform business decisions, whether built in-house or bought from vendors, regardless of technology. The Bank of England lists a revised version effective April 23, 2026.

In the US, the Treasury Department’s Office of Foreign Assets Control (OFAC) set out five components of an effective sanctions compliance program in its 2019 Framework for OFAC Compliance Commitments: management commitment, risk assessment, internal controls, testing and auditing, and training. Where technology supports controls, the framework calls for it to be tested and calibrated to the organization’s risk profile.

Those obligations stay with the regulated firm when it buys a vendor’s agent. Variation between runs of the same model, the pattern the benchmark measured, is the kind of result a validation exercise under either framework would be expected to examine.

Partnerships and the wider market

Castellum.AI was founded in September 2019 and raised an $8.5 million Series A led by CU Venture Fund-linked investor Curql in 2025. Piatetsky previously worked on Russia and Iran sanctions at the US Treasury Department.

Its LexisNexis Bridger integration places Arbiter’s decisions, explanations and audit records inside the screening platform. Piatetsky called it “literally putting our agent in the existing platform.” It has also announced plans to integrate its screening into Mintz Group’s Verity platform for enhanced due diligence, and a partnership with Hummingbird that keeps data, evidence and the audit trail in one case management system.

The benchmark lands as vendors across the market pitch agents for alert review, from Strise to established screening providers. Earlier RegTech Post coverage of AI advances in risk and compliance, unified KYC screening and AML resilience in digital systems covers the controls these tools sit within.

Castellum.AI will keep its detailed methodology available to approved parties on request, and the next test of whether the 500-case results hold is a client’s own closed-book run on unseen alerts.