Can a small local model match the flagship at finding PII?
A fine-tuned open detector against NVIDIA’s flagship and Microsoft Presidio, on the full 55-label vocabulary and the identical held-out records. Higher F1 is better; everything here runs locally, so no data leaves the machine.
Flagship accuracy, a third the size
On the full 55-label Nemotron-PII vocabulary, on a clean held-out test set, pii-proxy posts 91.5% fine F1 to NVIDIA gliner-PII’s 90.9% — from a base model a third the size (582 MB vs 1.7 GB), trained on a recipe you can rerun. The chart is accuracy against latency; smaller and faster is up and to the right.
Same records, both models
The full 55-label vocabulary, one clean held-out test split, the identical records fed to every model, threshold 0.5. Fine F1 is the head-to-head number; latency and weight size are the other two axes of the choice.
| Model | Fine F1 ↑ | Coarse F1 | Latency ↓ | Weights |
|---|---|---|---|---|
| pii-proxyDaslab | 91.5% | 93.9% | 144 ms | 582 MB |
| gliner-PIINVIDIA | 90.9% | 93.1% | 211 ms | 1699 MB |
| pii-proxy-bertDaslab · experimental | — | — | 26 ms | 438 MB |
| gliner_small-v2.1zero-shot | 54.8% | — | 115 ms | 582 MB |
On the healthcare slice the order reverses by a hair: gliner-PII 91.0%, pii-proxy 90.8%. pii-proxy-bert is a fixed-label BERT variant on a different (24-label) vocabulary — fast, but not part of this head-to-head; it’s here for the latency reference only.
Read the numbers by your constraint
Pick by your constraint
Flagship accuracy at a third the size, with zero-shot labels → pii-proxy (91.5% fine, 582 MB, 144 ms). The reference flagship → gliner-PII (90.9%, 1.7 GB, 211 ms). Sub-30 ms fixed-label → pii-proxy-bert (experimental).
Report the query strategy
GLiNER scores swing with the prompt: gliner-PII alone runs 79.8% / 90.4% / 97.2% on plain / natural / fine-label queries — the same weights. So a score without its query is noise. Both models here ran their best native-label query; the exact 55-label lists are in the repo.
Read this honestly
- Clean held-out test, leakage-checked. Both models scored on the same records from a
testsplit held out from training, with a zero-overlap assertion gating the run. An earlier eval on a train-holdout inflated these numbers by ~5 points — this guards against exactly that. - Same records, both queries published. One in-memory record set fed to both models, verified by hash; each ran its best native-label query at threshold 0.5, with span-overlap + exact-label matching. The label lists and the eval script ship in the repo.
- In-distribution measures reproduction, not generalization. Both models trained on Nemotron. The out-of-distribution result — 75.7% vs Presidio’s 42.0% — is the one that shows the fine-tune earning its keep.
# reproduce every number modal run experiments/modal_train.py --full --full-labels # fine-tune on the full 55-label Nemotron-PII python experiments/head_to_head_eval.py # score both models on the identical held-out records