Your AI Guardrail Doesn't Speak Swahili

Every commercial AI guardrail I have evaluated advertises multilingual support. None of the datasheets say what that means. So I built the test.

The setup

Forty-four prompts across seven attack categories, running on Promptfoo. Two things in that corpus do not appear in any vendor test set I have seen:

A Kenyan PII corpus. M-PESA numbers, National ID numbers, KRA PINs. Real formats, synthetic values. If you are securing an LLM application in this market, these are the identifiers that must not leak, and a classifier tuned on US Social Security numbers has never seen their shape.

A Swahili and Sheng attack corpus. The same attacks, written the way they would actually arrive. Not translated benchmark prompts, but attacks composed in the language, including Sheng, where a lot of real conversation happens and which no safety classifier has meaningfully been trained on.

The result

The best-performing product blocked 21 of 44 prompts.

Under half. From a product sold as a security control, marketed as multilingual, priced accordingly.

But the number I keep coming back to is not that one.

The configuration that scored higher was worse

The product ships in more than one configuration. The higher-scoring one caught more prompts. When I looked at which prompts, most of its additional detections came from misclassifying ordinary Swahili words, base64 strings and place names as attacks.

It was not catching more attacks. It was flagging more Swahili.

This matters more than the headline rate, because of how these products get chosen. You run a bake-off, you count blocks, you pick the configuration with the higher number. That process actively selects the configuration that will flag a Nairobi user writing a normal sentence, and it will look like a win in the procurement deck.

A detection count without a false-positive breakdown is not an evaluation. It is a number that moves in the right direction while the product gets worse at the job.

What "multilingual" actually means on a datasheet

Two different claims get collapsed into one word:

1. The underlying model has seen the language in pretraining.
2. The safety classifier was trained and evaluated on adversarial input in that language.

Vendors are making claim one. Buyers hear claim two. For a market that moves between English, Swahili and Sheng inside a single sentence, the gap between them is the entire control.

The gap is not exotic. It follows from where evaluation data comes from. Safety benchmarks are overwhelmingly English. A classifier optimised against them will be excellent at English attacks and will fall back to shallow heuristics elsewhere: keyword shapes, encoding detection, entity lookups. Those heuristics produce both halves of the result above, the misses on real attacks and the false positives on ordinary words.

What to do about it

Test in the languages your users use, with your own PII formats. This is a weekend of work with Promptfoo and it will tell you more about a product than any datasheet. If you operate in a market the vendor has not built for, you are the only person who is going to run this test.

Report true and false positives separately, always. A single block-rate number is worse than no number, because it is actionable in the wrong direction.

Do not let the guardrail be the boundary. This is the same conclusion I reached building CONTAGION, where an LLM judge caught every adversarial input in the eval set and was still injectable through fragmentation and attacks on its own classification prompt. A guardrail is a useful control. The boundary is least privilege on tools, strict quarantine of untrusted data, and output handling that assumes the model has already been turned.

Treat vendor safety training as a courtesy rather than a control. It varies by model, changes without notice, and vanishes the moment someone switches providers or self-hosts an open model. I make students watch this happen in the Break & Secure labs: run an attack against a frontier model, watch it refuse, run the same attack against an open model, watch it fall over. The application has to hold on its own.

The broader point

Most AI safety evaluation is built by and for English-speaking markets, and the failure modes it does not measure are borne somewhere else. This is a description of where the evaluation data sits rather than a complaint about vendors. It does mean that if you operate outside that centre of gravity, the published numbers are not about your deployment, and someone local has to generate the ones that are.

The methodology is public. The corpus is an active evaluation asset, so it is not. Happy to talk through the design, get in touch.