How AI content filtering actually works

"Content filtering" covers several quite different techniques. Knowing which one a product uses tells you more about what it will miss than any marketing page will.

Keyword and pattern matching

The oldest approach: a list of forbidden words and phrases, matched against the request. It is fast, cheap, completely predictable, and easy to audit — you can read the list.

It fails in both directions, badly. It misses anything phrased around the list, which takes a student roughly one attempt to discover. And it blocks legitimate work constantly: the classic example is a biology lesson that cannot mention anatomy. A filter that blocks real schoolwork trains students to route around it, which is worse than not having it.

Where it genuinely belongs is a short list of specific terms a school wants caught exactly, evaluated locally and instantly — not as the main line of defence.

Classifiers

A model trained to score text against categories — violence, sexual content, harassment. Better than keywords because it responds to meaning rather than spelling, so rephrasing does not defeat it.

The limits are that categories are fixed by whoever trained it, thresholds are a blunt dial, and it cannot express a rule like "this school does not discuss this particular subject". You get the categories on offer.

Model-based policy screening

The current approach: a separate model is given the policy in writing and asked to judge each request against it. Because the policy is text, it can be specific in a way a fixed category list cannot — including a particular school's own topics.

It is slower and costs a call per request, and it is not perfect: judgement calls near the boundary go both ways. What it buys is expressiveness. This is the only approach where "our school does not discuss X" is a rule the system can actually apply.

Before or after?

Two separate checks, and a serious product does both.

Screening the request before generation stops the obvious cases cheaply and means an unacceptable request never reaches a generating model at all. Screening the output after generation catches what the request did not reveal — an innocuous prompt can produce an unsuitable answer, particularly for images.

A product that only screens input is betting that a bad answer never follows a reasonable-looking question. That is not a safe bet.

The question that matters more than any of this

Every technique above depends on something that can be unavailable — a model, an API, a network hop. So the decisive question is what the system does when the check cannot run.

Fail open means the request proceeds unchecked, so the product stays up and the filtering silently stops. Fail closed means the request is refused until checking works again, so the filtering holds and the product is briefly unavailable.

A filter that fails open is not a filter. It is a filter-shaped feature that switches itself off exactly when nobody is watching.

For a tool used by children, failing closed is the only defensible choice, and it costs something real. Ask any vendor which they do and expect a straight answer.

And whose rules are they?

A technically excellent filter enforcing someone else's standard still is not your filter. The two questions to settle are whether you can add your own topics, and whether they apply to every tool rather than just the chat box.