How our document classifier votes before it ever asks a language model

A district council we work with has about 4 TB of files in SharePoint and more on internal SMB shares, and they need to know what is in them: which documents are contracts, which hold personal information, and which should have been disposed of years ago under the Public Records Act 2005. Every one of those files has to be read, classified into one of roughly 30 document types, and scored for sensitivity.

I built the classification layer of our Document Analyser to do that job, and the question I get most from other engineers is where the machine learning actually sits. The honest answer is that it is layered: classical NLP does most of the work, a transformer can join the vote, and a local LLM only gets a say when the others cannot agree. This post covers that pipeline. It does not cover the connectors, the compliance reporting, or the front end.

Text extraction and tokenisation come first

Every file is routed to a handler by MIME type. Office documents, PDFs, images, and audio each get their own. The handler extracts text (OCR for images), cleans it, and builds a preview that keeps section, sheet, and slide boundaries as break markers so a reviewer can see where a hit came from. The full text, not the preview, goes to the classifiers.

Tokenisation is NLTK. The classifier module bootstraps its own NLTK data directory inside the project and checks each required resource at import time, downloading it if missing, so a fresh container never fails on a missing tokeniser model halfway through a scan.

Both downstream classifiers consume the same token stream. The sensitivity detector combines NLTK tokens with a bank of regular expressions for personally identifiable information, financial details, credentials, and confidential business content, and returns a boolean, a numeric score, a level (low, medium, high), and the list of types matched. That detector is deliberately not a neural model. A regex for a credit card or bank account number is explainable, runs in microseconds, and does not hallucinate. When the output ends up in a privacy audit, explainable beats clever.

The document-type classifier is where the ML layering lives.

End-to-end classification path. Customer filename and content overrides win before the ensemble vote; scorer weights renormalise over whichever voters are active.

Three voters with different inductive biases

The classifier runs up to three scorers on the same text and treats each as a voter with a confidence in [0, 1]:

A keyword scorer: a curated dictionary of terms per document type, scored on token hits. Weight 0.2. Cheap, transparent, and brittle on documents that use unusual vocabulary.
A pattern scorer: regular-expression content patterns per document type. Weight 0.4. Captures structure the keyword list misses, such as how an invoice or a clause-numbered contract is laid out.
An optional language-model scorer, weight 0.4 when enabled. This is the transformer in the stack, loaded through Hugging Face transformers on PyTorch.

The three have different failure modes, which is the point. Keywords over-fire on boilerplate, patterns miss anything unstructured, and the transformer is slow and opaque but generalises. Weights are renormalised over whichever scorers are active, so with the LM off, keyword and pattern become 0.33 and 0.67.

Two overrides sit in front of the vote. A customer-defined filename pattern wins outright if it scores at least 0.3, and a customer-defined content pattern wins next. Councils have naming conventions no general model knows about, and honouring them beats guessing.

Combining the votes

Any scorer below 0.3 confidence, or returning unknown, is discarded. The survivors are combined per candidate type, in Python roughly:

score[t] = sum(conf[m] * w[m] for m in supporters[t])
score[t] /= sum(w[m] for m in supporters[t])
score[t] = min(1.0, score[t] * (1 + 0.1 * (len(supporters[t]) - 1)))

Dividing by the supporting weight means a type backed by one strong scorer is not penalised for the others staying silent. The 10 percent boost per extra supporter rewards agreement without letting three weak votes outrank one strong one. This is a hand-weighted soft-voting ensemble, and I am fine with that: with 30 classes and no labelled corpus at the start, a learned stacker would have been fitting noise.

The thresholds then decide. A top score at or above 0.80 is accepted. A top score between 0.50 and 0.80 is accepted only if it beats the runner-up by at least 0.15. Anything else is ambiguous or unknown, and only those files reach the LLM.

Decision flow after vote combination: accept if top score is at least 0.80; else accept if top is at least 0.50 with a 0.15 margin; otherwise optionally call a local Ollama LLM — full taxonomy for unknown files, shortlist-only for ambiguous ones — and keep the rule-based result if the model disagrees or memory pressure is high — When the ensemble is sure, it stops. The local LLM only sees unknown files (full taxonomy) or ambiguous shortlists, and cannot invent a type outside what the cheap scorers already proposed.

Every result records the type, the confidence, and a classification_method string (rule_based, filename_pattern, custom_pattern, ai_analyzer, size_limit), and all of it is persisted to MongoDB. That lets me aggregate average confidence per type and per method across a whole scan, which is the data you need in front of you before touching the 0.15 gap or the 0.3 floor.

The LLM as a constrained tie-breaker

The AI analyser is gated behind a USE_OLLAMA_FALLBACK setting and calls a local Ollama endpoint over HTTP. Nothing leaves the environment the scan runs in, which for a council handling ratepayer data under the Privacy Act 2020 is not optional.

Two things about how it is prompted matter more than which model is behind it. First, on an unknown file it sees the full taxonomy and its answer is accepted at 0.3 confidence or better. Second, on an ambiguous file it is passed only the shortlist of candidate types the cheap scorers produced, and its answer is accepted only if it agrees with the top or second choice at better than 0.50. A model that picks something neither scorer suggested is ignored. That constraint is the difference between a tie-breaker and a second opinion nobody can audit.

There is also a memory guard that reads the container's cgroup limit and skips the model call above 85 percent usage, falling back to the rule-based result. It has its own test suite, but it is plumbing, not ML, so I will leave it there.

What I would tell a peer starting the same build

The Office of the Privacy Commissioner received 1,093 breach notifications in the year to June 2025. Most NZ organisations I talk to cannot answer "where is the personal information in our file shares" without a scan like this.

A few things I would do again, and one I would not:

Keep sensitivity detection deterministic. Auditors and privacy officers need to see why a file was flagged.
Record the classification method and confidence on every file. Being able to query "everything the AI analyser decided" is how you tune thresholds against real data instead of intuition.
Give each voter a different bias. An ensemble of three keyword lists is one keyword list.

The one I would change: the weights and thresholds are still hand-set constants. Users can already correct a document type in the UI and the correction is stored. That is a labelled dataset accumulating for free, and the next step is to fit the stacker on it, or at minimum to fine-tune the transformer scorer per customer.

If you are starting the same build, begin with the tokeniser and the deterministic scorers, and add the transformer and the LLM only once you can measure what they change. Happy to compare notes if you get stuck on any of it.