Invarix.Guard vs DeBERTa-v3 and Microsoft Presidio: a reproducible head to head benchmark

We benchmarked Invarix.Guard against the strongest open source baseline for each task:

• DeBERTa-v3 prompt injection model. ProtectAI/deberta-v3-base-prompt-injection-v2, the open source SOTA prompt injection model and the one LLM Guard's PromptInjection scanner wraps internally. The same model is the prompt injection backbone inside Invarix.Guard.
• Microsoft Presidio. The de facto open source PII analyzer, spaCy NER backbone.

Three benchmarks across 2,450 multilingual prompts and 210 synthetic PII rows. All datasets are public, and the methodology is described in full below.

Summary

Combined headline: Beats Microsoft Presidio on PII detection F1 with zero false positives. Matches the open source SOTA DeBERTa-v3 prompt injection model on detection metrics at comparable latency. Survives adversarial perturbation as well as the dedicated comparator. All in process .NET. No Python sidecar.

Why this benchmark

ProtectAI/deberta-v3-base-prompt-injection-v2is the open source SOTA prompt injection model. It is what LLM Guard's PromptInjection scanner wraps internally. Microsoft Presidio is the de facto open source PII analyzer. We wanted to know: does Invarix.Guard, which embeds these capabilities in process for .NET applications, hold up against the dedicated tools when run head to head?

This isn't a benchmark against a closed commercial product we can't reach. It's a benchmark against the strongest open source baselines for each task, run on the same input we'd run through Invarix.Guard. If you're evaluating commercial guards, run them on the same fixture: the harness, per-row results, and rerun commands are public.

Methodology

Scoring.Binary. A guard “flags” an input when it returns Block or Flag (Invarix), is_valid: false (LLM Guard sidecar), or any entity above 0.5 confidence (Presidio). TPR = flagged / actual positives. FPR = flagged / actual negatives.

1. Multilingual prompt injection

Datasets (all public). 200 English attacks from HumanCompatibleAI/tensor-trust-data + 900 multilingual attacks from walledai/MultiJail (ICLR 2024) covering Arabic, Bengali, Italian, Javanese, Korean, Swahili, Thai, Vietnamese, Chinese (100 each) + 300 NLLB-200 translated TensorTrust extraction attacks in German, Spanish, French, Japanese, Portuguese, Russian (50 each) + 150 PromptBench style perturbed extraction attacks (see §2). Clean prompts: 200 English from databricks/databricks-dolly-15k + 700 multilingual from OpenAssistant/oasst1. Total: 2,450 rows, 16 languages.

Comparator: ProtectAI/deberta-v3-base-prompt-injection-v2 loaded via HuggingFace transformers in a small FastAPI sidecar.

Detection metrics are deterministic and reproduce byte identical across runs. Latency is system dependent. Across two runs on the same fixture, Invarix's P50 ranged 129-189ms, DeBERTa's 226-256ms. Take the latency story as “comparable,” not a headline win.

Per language attack catch rate

Where both detectors are strong: the NLLB translated extraction set (de/es/fr/ja/pt/ru) lands at 94-100% TPR for both guards. Translation preserves enough injection signal that detection holds up. Bengali (98/100) is the only professionally translated MultiJail language where both detectors stay strong. Everywhere else outside the NLLB set, performance drops sharply.

Where both detectors flop:Italian (18/15), Chinese (4/5), Korean (22/26), Vietnamese (35/38), Swahili (41/41), and Javanese (48/43). All the MultiJail multilingual jailbreak attacks land far below the comparator's English performance, at comparable rates between the two guards. Arabic (67/75) and Thai (87/95) are middling. This is the detection technology's actual ceiling, not an Invarix limitation.

Per language false positive rate

OASST1 is volunteer collected and not fully curated. There is circumstantial evidence that some of the “clean” Spanish / French / Portuguese rows contain unlabeled jailbreak attempts (both guards flag the same rows, suggesting they are real attacks rather than independent FPs). The headline numbers don't try to correct for that.

2. Adversarial robustness

We applied three PromptBench style perturbations to 50 English TensorTrust extraction attacks each, generating 150 adversarial rows:

• TextBugger. Random character insertions / deletions / swaps + Cyrillic homoglyph substitutions (10% of letter chars).
• WordDeletion. Drops 20% of non stopword tokens.
• CaseFlip. Randomly upper/lowercases 30% of letters.

Both detectors are perturbation robust. Only a 3-4pp drop from the original 97-98% catch rate even under character edits + Cyrillic homoglyph injection + 20% word deletion. CaseFlip didn't move the needle (regex case insensitive + ML normalizes case anyway). The multi task scanner architecture isn't brittle to evasion.

3. PII detection (vs Microsoft Presidio)

Fixture:111 synthetic PII rows + 99 clean rows = 210 total. PII types covered: EMAIL, PHONE, SSN, CREDIT_CARD, IP, NAME, ADDRESS, DATE_OF_BIRTH. All “real looking” PII values are fabricated: RFC 2606 example.com domains, 555 area code phones, 192.0.2.0/24 IPs, IIN-9999 Luhn valid credit cards, 9XX area SSNs. Fixture seed = 42 for reproducible generation.

Comparator: Microsoft Presidio (presidio-analyzer 2.2.362), spaCy en_core_web_lg backbone, via local FastAPI sidecar.

Invarix wins on F1 (+3.1pp), accuracy, and zero false positives vs Presidio's 12%. Presidio wins on latency by an order of magnitude (~11× faster). It is a single task NER pipeline, while Invarix runs the full guard stack (PII + injection + toxicity + safety) on every input. Disable the scanners you don't need via GuardOptions.EnablePiiDetection / EnableContentSafety / EnablePromptInjection and the gap closes.

Per PII type detection rate

The DATE_OF_BIRTH gap is by design in Invarix. Bare dates require a “born / dob / birthday” context word to flag, avoiding false positives on log lines, calendar entries, and historical dates. Presidio flags any date shaped string. Real world tradeoff: Invarix's design avoids last seen 03/15/2024 triggering PII alarms, while Presidio catches all dates, including innocent ones. The 12% FPR difference is largely from this.

A note on latency

Invarix.Guard's P50 (189ms on the prompt injection benchmark, 109ms on the PII benchmark) is the cost of running four ML scannerson every input: prompt injection (DeBERTa-v3), PII NER (XLM-RoBERTa multilingual), embedding based content safety (e5-base), and toxicity (multilingual DistilBERT). The single task comparators reflect their narrower scope. The DeBERTa sidecar (226ms P50) is slowed by the IPC hop and PyTorch's eager mode inference. Presidio (10ms P50) is faster because spaCy NER is leaner.

The disable knob matters.Most production deployments don't need all four scanners. GuardOptions.EnablePromptInjection / EnablePiiDetection / EnableContentSafetytoggle the heavy work off when you don't need it, and latency drops roughly linearly with the number of scanners enabled.

We tried parallelizing the four ML scanners via Parallel.Invoke(independent ONNX sessions, each thread safe per Microsoft.ML.OnnxRuntime docs). Across multiple measured runs the change netted roughly zero on aggregate. Threadpool dispatch overhead matched the parallelism win when individual scanners were already fast (clean inputs, ~30-100ms each). The slow tail is concentrated on long adversarial inputs where SentencePiece tokenization runs O(n) regardless of inference, so reordering doesn't help.

The architectural fixes that would actually move P95 are bigger projects: a multi task model (one inference for all scanners), smaller or quantized backbones, and token level input capping. They are on the roadmap, not in v1.2.

Caveats and known limitations

• EU language attack data is machine translated. NLLB-200-distilled-600M preserves injection intent but may not reflect the natural distribution of native speaker attacks.
• Multilingual ATTACK coverage is uneven. MultiJail's languages don't overlap with OASST1's clean prompt languages. Only Chinese has both, so the per language F1 calc only works there.
• OASST1 contains some unlabeled jailbreak attempts in non English splits, inflating the FPR for both guards on those languages.
• n=50-100 per language is small. Confidence intervals at this size are ±5pp. Per language differences within ±3pp should be read as ties.
• PII fixture is synthetic. Hand crafted templates with deterministic fakes. Real world PII patterns may differ.
• The same DeBERTa model is wrapped in both prompt injection pipelines. This is a benchmark of the integration around the model, not of which model is better.
• Latency is system dependent. Same fixture, same machine, two runs on different days produced P50 differences of ~30% on the prompt injection benchmark. Detection metrics, by contrast, reproduced byte identical. Treat the latency numbers above as one data point, not a precise measurement.

Run it yourself

Every detection number above reproduces from the committed harness: fixtures, per-row result JSON, and rerun commands are in the public repo under benchmarks/. If you want the same stack in your own process, the Community tier is free: dotnet add package Invarix.Guard. It runs in process in your ASP.NET Core app, and nothing you scan leaves the box.