PII removal with local SLMs

Nebuly analyses conversations. Our customers run assistants and agents in production, and we tell them what their users are actually asking for, where the product is failing them, and what to fix next. The raw material is conversation logs.

Conversation logs are also one of the worst places personal data ends up, because users type whatever they type. Under current privacy policy most of our customers cannot hand those logs to us until the personal data is out of them. So PII removal is not our product. It is the gate standing in front of our product, and for a while it was the most common reason a deployment took months instead of weeks.

What made it slow is that nobody agrees on what has to go. "PII" sounds like a fixed list and it is not: it depends on the data and on who is looking at it. A bank has to strip every identifier that can be tied back to a customer, and that reaches past names and account numbers down to a balance or a transaction amount distinctive enough to single somebody out. An investment firm may not care about any of that, and may instead need the names of its portfolio companies obscured, because the sensitive fact is not who the user is but which deals are being discussed. A support team exporting tickets frequently wants the opposite of both: names and emails gone, cities and dates kept, because the geography and the timeline are the analysis.

So the class list cannot be something we fix at training time. It has to be an argument the caller passes in, in their own words, per request. That constraint is what rules most things out.

It points at an LLM, and it points straight at a problem. The most capable models are hosted, and to have a hosted model strip your personal data you first have to ship it your personal data, unredacted. For the customers who need this most, that is not a trade-off to be priced. It is a closed door.

And for everyone else there is the bill. We are not redacting a corpus once. We are redacting every conversation a product handles, continuously, for as long as the customer is a customer, which makes redaction a line item that scales linearly with product usage and never goes away. At frontier-model list price it costs roughly eight times what the same job costs on a model we host ourselves, and about twenty times once the model is warm. Full numbers in the cost section below.

Which leaves the question this post is about: how good can a model be if it has to run on one GPU inside the customer's own network? We built a 1,396-document multilingual benchmark with a 21-class caller-defined schema and ran four systems against it: our own 8B model, gpt-5.6-sol as a ceiling, Gemma 4 E4B as a same-size control with no task-specific training, and Azure AI Language as the managed-service baseline.

The short answer is that this task saturates well below frontier scale. Out of 3,027 PII values the frontier model missed 78 and ours missed 80, with nothing leaving the network. The control is the part worth keeping in mind: the same parameter count without task-specific training missed thirteen times more.

The rest of this post is the evidence, in three parts. First the dataset, which is deliberately unlike a public NER benchmark: ten languages, twenty-one caller-defined classes, values corrupted on purpose, and a lot of documents with little or no PII in them at all. Then the four systems and how the comparison was set up, including what it does not measure. Then the results, category by category and language by language.

The dataset

Everything below runs on an internal evaluation set we built to test our own models and compare them against the alternatives. It is 1,396 synthetic business documents: CRM contact updates, sprint retrospectives, support threads, invoicing notes, HR records. The kind of text a redaction pipeline actually sees, not the news articles and Wikipedia passages most public NER benchmarks are built from.

Four things set it apart from a public NER benchmark.

It is genuinely multilingual, not English with translations bolted on. Ten languages, roughly 140 documents each: English, Spanish, French, German, Italian and Portuguese in Latin script, then Arabic, Hindi, Chinese and Japanese. Four of the ten use a non-Latin script, which is where off-the-shelf detectors tend to fall apart: their tokenisers, gazetteers and regexes were all built for Latin text.

The taxonomy is wider than a standard PII list. Twenty-one types, where a conventional redaction setup ships five or six. The breadth follows directly from the disagreement described above: adding a class has to cost a name and a definition, not a retraining run. It is also why the set splits categories a coarser model would lump together. A full card number is its own class, separate from the last four digits; a city is separate from a country, and a street from a full address.

The PII is deliberately corrupted. Real documents are not clean. People type names in lower case, paste ALL-CAPS surnames out of a legacy CRM, write emails as name[at]example.org to dodge scrapers, fat-finger a digit in a card number, and hand you OCR output with a transposed letter. A detector tuned on tidy synthetic PII looks excellent in the lab and then collapses on exactly this. So a share of every category is perturbed on purpose: casing destroyed on 35% of proper nouns, 31% of full card numbers failing a Luhn check, 16% of email addresses malformed or obfuscated, plus character-level noise (ipazza alloisio, alexanderhavrn, ll ucc.iano@example.com) and implausible values like a birth year of 9182. Detecting the corruption after the fact flags about one value in ten, which is a floor: single-character edits inside correctly-cased text are invisible to that check.

Most of the text is not PII. PII accounts for 5.3% of the corpus by character count. The average document runs about 960 characters and contains one PII value every 63 words. Thirteen percent of the documents contain no PII at all, 44% contain zero or one value, two thirds contain two or fewer, and only 10.5% contain five or more.

The shape of that second panel is the point. The most common document in the set carries exactly one PII value, and the distribution thins out fast: only 147 documents reach five values or more. The benchmark is therefore made mostly of sparse text rather than PII-dense forms, and that is what makes over-redaction measurable at all. A system judged only on dense text looks good by flagging aggressively, because there is nearly always something there to find. Hand it the 183 documents that contain nothing and the same instinct rewrites them anyway. We come back to this in the analysis, where it is the axis that most cleanly separates a cautious system from an aggressive one.

The left panel is the other deliberate choice. The 21 types are close to uniform by construction, every one carrying between 121 and 174 values, a spread of just 1.4x. That is not what a production stream looks like, where names, emails and phone numbers dominate and the tail types (a postal code on its own, a bare month, the last four digits of a card) are rare. Flattening the curve cuts both ways: it raises the weight of the hard, ambiguous categories and lowers the weight of the easy ones, so every absolute score in this post is lower than the same system would post on production-like data. Read them as a comparison between methods, not as a forecast for your own corpus.

Table 1: The dataset at a glance

Table 3: PII taxonomy, 21 annotated types into 9 evaluation categories

Company addresses are counted as location, not company. That choice is what makes location the largest category, and therefore the largest false-positive source for two of the four systems we tested.

Design choices and constraints

Each record has three fields, and together they define the interface every system in this benchmark has to implement. The first two are the input: the document text, and the schema of classes to look for. The third is the output the model is asked to produce, and in the dataset it holds the ground truth we score against.

{
  "text": "Kontakt-Update für den CRM-Datensatz\n\nNachname: Veer\nStadt: backnang\nGeschlecht: transgender\nZusätzliche Geschlechtsangabe: DIVERS\nKunde seit dem Jahr: 2010\n...",

  "pii_definitions": {
    "LAST_NAME": "The family name of any person mentioned.",
    "CITY":      "The name of a city, town, municipality, or village.",
    "YEAR":      "Any year value mentioned, regardless of context.",
    "GENDER":    "A person's gender identity or sex designation.",
    "...":       "(21 classes in total)"
  },

  "pii_mask": {
    "Veer":        "LAST_NAME",
    "backnang":    "CITY",
    "transgender": "GENDER",
    "DIVERS":      "GENDER",
    "2010":        "YEAR"
  }
}

{
  "text": "Kontakt-Update für den CRM-Datensatz\n\nNachname: Veer\nStadt: backnang\nGeschlecht: transgender\nZusätzliche Geschlechtsangabe: DIVERS\nKunde seit dem Jahr: 2010\n...",

  "pii_definitions": {
    "LAST_NAME": "The family name of any person mentioned.",
    "CITY":      "The name of a city, town, municipality, or village.",
    "YEAR":      "Any year value mentioned, regardless of context.",
    "GENDER":    "A person's gender identity or sex designation.",
    "...":       "(21 classes in total)"
  },

  "pii_mask": {
    "Veer":        "LAST_NAME",
    "backnang":    "CITY",
    "transgender": "GENDER",
    "DIVERS":      "GENDER",
    "2010":        "YEAR"
  }
}

{
  "text": "Kontakt-Update für den CRM-Datensatz\n\nNachname: Veer\nStadt: backnang\nGeschlecht: transgender\nZusätzliche Geschlechtsangabe: DIVERS\nKunde seit dem Jahr: 2010\n...",

  "pii_definitions": {
    "LAST_NAME": "The family name of any person mentioned.",
    "CITY":      "The name of a city, town, municipality, or village.",
    "YEAR":      "Any year value mentioned, regardless of context.",
    "GENDER":    "A person's gender identity or sex designation.",
    "...":       "(21 classes in total)"
  },

  "pii_mask": {
    "Veer":        "LAST_NAME",
    "backnang":    "CITY",
    "transgender": "GENDER",
    "DIVERS":      "GENDER",
    "2010":        "YEAR"
  }
}

{
  "text": "Kontakt-Update für den CRM-Datensatz\n\nNachname: Veer\nStadt: backnang\nGeschlecht: transgender\nZusätzliche Geschlechtsangabe: DIVERS\nKunde seit dem Jahr: 2010\n...",

  "pii_definitions": {
    "LAST_NAME": "The family name of any person mentioned.",
    "CITY":      "The name of a city, town, municipality, or village.",
    "YEAR":      "Any year value mentioned, regardless of context.",
    "GENDER":    "A person's gender identity or sex designation.",
    "...":       "(21 classes in total)"
  },

  "pii_mask": {
    "Veer":        "LAST_NAME",
    "backnang":    "CITY",
    "transgender": "GENDER",
    "DIVERS":      "GENDER",
    "2010":        "YEAR"
  }
}

{
  "text": "Kontakt-Update für den CRM-Datensatz\n\nNachname: Veer\nStadt: backnang\nGeschlecht: transgender\nZusätzliche Geschlechtsangabe: DIVERS\nKunde seit dem Jahr: 2010\n...",

  "pii_definitions": {
    "LAST_NAME": "The family name of any person mentioned.",
    "CITY":      "The name of a city, town, municipality, or village.",
    "YEAR":      "Any year value mentioned, regardless of context.",
    "GENDER":    "A person's gender identity or sex designation.",
    "...":       "(21 classes in total)"
  },

  "pii_mask": {
    "Veer":        "LAST_NAME",
    "backnang":    "CITY",
    "transgender": "GENDER",
    "DIVERS":      "GENDER",
    "2010":        "YEAR"
  }
}

The input is a document plus a schema. pii_definitions is not metadata about the dataset, it is an argument. The classes to look for, and the natural-language definition of each one, arrive at inference time as data. They are not baked into the model weights and not fixed at training time.

The output is a mapping from detected value to class. Not character offsets, not an annotated copy of the text: a dictionary whose keys are the literal substrings the model believes are PII. Redaction is then a substitution pass over those keys.

That is a hard requirement, not a stylistic choice. The model has to take a subset of the classes and mask only those: nothing outside the requested schema, nothing inside it skipped. It also has to handle classes it has never been trained on, defined on the spot in prose, which is why the definitions read as instructions rather than bare labels.

We test the wording too. Every one of the 21 classes ships five different paraphrases of its definition, sampled independently per document, so all 1,396 records carry a unique schema instance. CREDIT_CARD_LAST_DIGITS, for example, appears variously as "Partial card numbers consisting of just the last four digits", "Only the final four digits of a credit or debit card, not the full number", and "Last-four references used for identity verification (e.g. 'ending in 4242')". A model that has quietly memorised one exact phrasing scores worse than a model that reads the definition it was given.

Because the schema travels with each request, one deployment serves a customer who wants three classes and a customer who wants all twenty-one, with nothing to retrain or redeploy in between. That is the design choice; the benchmark below only exercises the maximal case.

The output format also sets the unit of accounting, and that changes how every number below should be read. Since the keys of pii_mask are values rather than positions, the unit of evaluation is a unique value, not a text occurrence. The 1,396 documents hold 3,027 unique values, which appear 5,763 times in the text, and 641 of the 1,213 documents that contain PII repeat at least one value. A system that returns "2010": "YEAR" is credited once, no matter how many times 2010 occurs. That is the right unit for scoring detection; the substitution pass afterwards is what has to catch every occurrence. Every recall, leak-free and exact-match figure below is value-level.

One constraint follows, and it rules out a whole class of tool. A schema supplied at inference time is something a generative model can accept and a fixed-taxonomy service structurally cannot: Azure AI Language has a closed, vendor-defined entity list, so there is simply nowhere to put pii_definitions at all. It cannot be told which classes matter, which is a different and more basic limitation than being inaccurate. How we handled that in the comparison is in the test setup below.

Methodology: four ways to remove PII

We did not pick four models. We picked four positions in the decision space a team actually faces, then the strongest representative of each. Two questions separate them: where the compute runs, and who decides what counts as PII.

Nebuly SLM (8B), self-hosted. A small language model fine-tuned for schema-driven PII extraction: it takes the document and the pii_definitions schema, and returns the value-to-class mapping described above. It runs inside the perimeter on a single GPU, served with vLLM, one document at a time. This is the system currently serving Nebuly Enterprise customers in production.

Gemma 4 E4B (8B), self-hosted. The control, and the measure of what the fine-tuning is worth. Same parameter budget, same GPU, same serving stack, same documents, same schemas, no task-specific training. One detail worth volunteering, because it cuts against the easy objection that we simply picked a stronger starting point: the open model we fine-tuned scores below Gemma 4 E4B before any training, and a fine-tuned Gemma still lands behind it afterwards on our internal benchmarks. Which model responds best to training is a different question from which model is best untouched, and it is the subject of a separate post.

gpt-5.6-sol (Azure OpenAI). The ceiling, not a competitor. A frontier reasoning model given the same document and the same schema, one document per request, with reasoning tokens billed. Its job in this benchmark is to answer a single question: how good can schema-driven PII removal get when neither cost nor data residency is a constraint? A ceiling rather than a candidate, for the reasons in the introduction. We measure it anyway: a bar you cannot use is still the right bar to be measured against.

Azure AI Language PII detection. The managed-service baseline: what you get by calling the box that is already sitting in your cloud subscription. It has a closed, vendor-defined entity list and no way to accept a schema. We include it because it is what a lot of teams reach for first, and because the comparison is not really about accuracy: it is about whether "what counts as PII here" is a decision you make or one that is made for you.

Test setup, and what it does not cover

All four systems saw the same 1,396 documents, each paired with the same per-document schema, and returned the same value-to-class mapping. One scorer read all four outputs.

We score in two modes. Span matching asks only whether the system found the right piece of text; typed matching also requires the right class. Unless stated otherwise every number in this post is typed, which is the stricter of the two.

Azure AI Language needs a translation layer, since it cannot be given our schema. Its native entity types map onto eight of our nine categories: Person to person, PersonType to gender, Address to location, DateTime to date, Organization to company, and so on. Nothing it returned mapped onto monetary amounts, which is why it scores 0.00 in that category. Four of its labels have no counterpart in our taxonomy at all (URL, IPAddress, USSocialSecurityNumber, USDriversLicenseNumber), 89 predictions across the run. We exclude those rather than charge them as errors: a system should not be marked down for answering a question nobody asked. It barely matters either way, counting them moves Azure's F1 from 0.352 to 0.349.

Analysis

TLDR; the task has a ceiling, an 8B model reaches it, and reaching it is about training rather than size. Below is the evidence, including where the frontier model is genuinely ahead.

Table 4: Headline results

Leak-free is the share of documents where every real PII value was caught. Exact is the share where the redaction matched the ground truth perfectly, nothing missed and nothing extra. Speed and cost have a section of their own further down.

Parity on the number that decides whether data leaks

Of the 3,027 PII values in the corpus, gpt-5.6-sol left 78 uncaught. Our 8B model left 80. Two values, out of three thousand, separate a frontier reasoning model from something that runs on one GPU in your own rack. At the document level the same thing shows up as 95.2% of documents leak-free against 95.1%, which is a difference of one document in 1,396.

That is the whole argument in one comparison. It does not say the two systems are equivalent. It says that on the question "did any personal data survive the pass", frontier scale bought nothing measurable. Recall is 0.974 for both.

The two hosted alternatives are not close. Azure AI Language leaves 1,564 values uncaught, nearly twenty times as many as either strong system, and only 37.0% of its documents come out clean. Gemma 4 E4B leaves 1,051, thirteen times as many, at 56.0% leak-free.

Why we lead with leak-free and not F1

Look at Chart 01 and the F1 column tells a much duller story: 0.79 for gpt-5.6-sol, 0.78 for us, 0.70 for Gemma, 0.35 for Azure. It also hands the best precision of all four systems to Gemma, at 0.75.

Both are artefacts of averaging two errors that are not comparable. A missed value is a disclosure: once the corpus has been shared, handed to a vendor, or used for training, it cannot be taken back. An over-redaction is a loss of utility: annoying, measurable, and fixable by re-running with a narrower schema. Averaging them into one number treats a leak as interchangeable with a redundant black box, and it produces exactly the wrong ranking. Gemma is the most precise system in the benchmark and the least safe of the three language models. Optimising precision is optimising for the wrong failure.

So we report the two separately and lead with the one that carries the risk. In Chart 02, Nebuly and gpt-5.6-sol sit at the same height and separate horizontally: the shape of a utility difference, not a safety difference.

Same size, opposite outcome

Nebuly's model and Gemma 4 E4B are both 8B parameters, ran on the same GPU with the same serving stack, and received the same documents and the same schemas. Recall: 0.974 against 0.653. Leak-free documents: 95.1% against 56.0%.

The gap is not concentrated in some exotic corner. It is everywhere, and it is enormous:

This is the result to point a sceptic at, because it rules out the cheap reading of the headline. "A small model is enough" is not what this run shows. What it shows is that a small model trained for this task is enough, and that an equally small model that has only been told what to do falls short by a factor of thirteen on the thing that matters. Email is the one category where prompting alone gets close, at 0.91, and email is also the one category with an unambiguous surface form. Everywhere the decision needs context, the untrained model loses.

The multilingual result holds

The usual objection to a small model is that it will hold up in English and quietly degrade everywhere else. It does not happen here. Across all ten languages our recall stays between 0.948 and 0.985, and gpt-5.6-sol sits in the same band at 0.951 to 0.994.

Gemma is the interesting contrast again: its recall swings from 0.58 in German to 0.74 in Japanese, so the untrained model is not only worse on average but less predictable per language, which is worse still for anyone who has to sign off on a data-protection review.

Table 9: Latin-script languages vs non-Latin-script languages

859 documents in the six Latin-script languages, 537 in the four non-Latin-script ones.

• Table 8: the full per-language breakdown

  Recall by language

  System	En	Es	Fr	De	It	Pt	Ar	Hi	Zh	Ja
  Nebuly SLM (8B)	0.98	0.97	0.98	0.98	0.98	0.97	0.98	0.95	0.98	0.97
  gpt-5.6-sol	0.99	0.98	0.98	0.98	0.96	0.96	0.95	0.99	0.98	0.98
  Gemma 4 E4B (8B)	0.70	0.73	0.66	0.58	0.63	0.65	0.59	0.64	0.60	0.74
  Azure AI Language	0.52	0.47	0.51	0.56	0.43	0.49	0.47	0.43	0.45	0.50

  Precision by language

  System	En	Es	Fr	De	It	Pt	Ar	Hi	Zh	Ja
  Nebuly SLM (8B)	0.63	0.69	0.73	0.67	0.60	0.62	0.62	0.62	0.68	0.70
  gpt-5.6-sol	0.66	0.67	0.63	0.65	0.68	0.64	0.64	0.64	0.72	0.73
  Gemma 4 E4B (8B)	0.72	0.80	0.75	0.66	0.80	0.68	0.74	0.65	0.85	0.91
  Azure AI Language	0.23	0.31	0.26	0.33	0.24	0.28	0.32	0.31	0.23	0.28

Precision is flat across languages too, for every system. The over-flagging does not concentrate in any language. It concentrates in particular categories, which is where the next section goes.

The remaining gap is one category wide

gpt-5.6-sol does beat us on one real thing: 70.2% of its documents come out exactly right against our 54.4%, a sixteen-point gap. Here is where it comes from.

Chart 03 shows our model at 0.98 or better on person, location, date, credit card and phone, and at 1.00 on gender, and then a single cell at 0.17 for monetary amounts. Chart 06 explains it: of our 1,570 false positives, 1,342 are monetary amounts, 85% of the total. Against a support of only 136 real money values, the model flags about eleven numbers for every one that is actually a sum of money. It finds essentially all the real ones, recall 0.99, and drags along everything numeric that could plausibly be a price.

Chart 07 shows what happens if that one category is set aside. Our F1 goes from 0.78 to 0.95; gpt-5.6-sol, with its own worst category removed, goes from 0.79 to 0.86. The metric flatters whoever has the most concentrated errors, and here that is us, so read it as "one rule or a general weakness?" rather than as a ranking. gpt-5.6-sol has the same shape of problem spread over two categories instead of one: 62% of its false positives are locations, and its location and date F1 are 0.67 and 0.69, which is why it over-flags at almost exactly our rate overall.

Table 7: Error profile and error concentration

• Table 6: F1 by PII category, the numbers behind Chart 03

  System	Person	Email	Phone	Credit card	Location	Date	Company	Money	Gender
  Nebuly SLM (8B)	0.98	0.79	0.99	0.99	0.98	0.98	0.66	0.17	1.00
  gpt-5.6-sol	0.91	0.99	1.00	1.00	0.67	0.69	0.95	0.95	0.98
  Gemma 4 E4B (8B)	0.74	0.95	0.82	0.84	0.58	0.72	0.77	0.60	0.74
  Azure AI Language	0.79	0.87	0.61	0.39	0.46	0.33	0.35	0.00	0.05

  Support per category: location 992, date 598, person 411, credit card 297, email 152, gender 152, company 150, phone 139, money 136.

That is the substantive finding of the benchmark. All four systems have one or two categories where the errors pile up, and it is a different set each time. None of it is visible in an aggregate F1, and all of it is actionable: a category-shaped failure is fixed with category-shaped training data.

Nobody asked for the clean documents to be rewritten

The 183 documents with no PII are the cleanest test in the set: the correct output is the input. Nebuly leaves 90.2% of them untouched, gpt-5.6-sol 90.7%, Gemma 93.4%. Azure AI Language leaves 8.2%: it altered 168 of the 183, adding 453 redactions to documents that needed none, about two and a half per document.

Table 10: The over-redaction test, 183 documents with no PII

This bounds the over-flagging problem. It is real, but it is not a general tendency to redact anything that moves: on text with nothing to find we are within half a point of the frontier model. The money false positives are triggered by numbers that look like amounts, not by a blanket bias toward flagging.

The economics of keeping it in house

Priced on wall-clock time on one Azure A100 at $4.591 an hour, the whole 1,396-document run cost $0.35 on our model. The same run through gpt-5.6-sol cost $2.82 at list price, reasoning tokens included. Table 13 has the per-document breakdown. $0.25 per 1,000 documents against $2.02, and $0.10 once the one-off model load is excluded, which no production service pays repeatedly.

Eight to twenty times cheaper at the same leak-free rate, on a cost that recurs for every document you will ever process.

Table 13: Cost model

• Table 11: throughput and latency in detail

  System	Wall time	Model load	Inference	Docs/s end to end	Docs/s warm	Latency p50	Latency p95	Requests
  Nebuly SLM (8B)	271.2 s	163.5 s	105.9 s	5.1	13.2	not applicable	not applicable	not applicable
  gpt-5.6-sol	246.1 s	not applicable	not applicable	5.7	5.7	2,735 ms	7,242 ms	1,396
  Gemma 4 E4B (8B)	189.9 s	113.0 s	71.5 s	7.4	19.5	not applicable	not applicable	not applicable
  Azure AI Language	22.4 s	not applicable	not applicable	62.3	62.3	339 ms	775 ms	280

  Concurrency is not matched across backends, so the throughput column is not apples-to-apples: gpt-5.6-sol ran about 19 requests in flight, Azure AI Language about 5 in batches of five, and the two self-hosted models one document at a time.

Gemma looks cheaper still at $0.17, and that number is misleading. It ran inference in 71.5 seconds against our 105.9, but emitted 2,646 predictions against our 4,517: two thirds of the time for half the output. Per prediction it is slower than we are, 27.0 ms against 23.4 ms. Its lower bill is bought by not finding the PII.

One number does go the other way. Azure AI Language is twelve times faster end to end, 62.3 documents a second against our 5.1, because it is a purpose-built classifier rather than a generative model. If throughput were the only constraint it would win comfortably. It finds under half the PII.

What this says about small models

The gate we described at the start is cheap enough to stop thinking about. An 8B model on one GPU inside the customer's network misses 80 PII values out of 3,027, two more than a frontier reasoning model, at an eighth of the cost, with nothing leaving the perimeter. The bank, the investment firm and the support team from the opening all get their own class list out of the same deployment, because the schema is an argument and not a build.

That is a claim about this task, not about model size in general. But the shape of the task is worth naming, because a lot of production work has it:

• the decision is local: one short document in, no state to carry across calls

• the output is small and structured: a list of values and labels, not prose

• the specification arrives at call time: the schema is data, so behaviour changes without retraining

• the volume is high and recurring: unit cost is the whole economics

• the data is sensitive: where inference runs is part of the requirement

When those hold, a fine-tuned SLM is not a compromise. It reaches the same ceiling, costs eight to twenty times less per document, runs where the data already is, and leaves you owning the weights instead of renting an endpoint. A frontier model is how you find out where the ceiling is, and what you use while the task is still moving. Paying for it in steady state, on work of this shape, is renting capability you have already measured and do not need.

The counter-example is in the same benchmark, which is why we ran the control. Gemma 4 E4B: same size, same GPU, same inputs, thirteen times more PII missed. Small is not the point. Small and adapted is.

Nebuly

Nebuly is the ROI platform for enterprise AI. It connects to the AI agents your business runs on, the assistants your customers interact with, and the tools your employees use every day, including Claude, ChatGPT, and Copilot, and translates that activity into business value. How much time is being saved across teams. What revenue your AI is influencing. What adoption and AI proficiency look like in practice, across departments and geographies. All aggregated at the organizational level, never tied to individuals.

If you need clarity on what your AI investment is actually delivering, book a demo.