The annotator that never tires: making data useful with AI

A customer database can contain years of correspondence and still be unable to answer a simple question: which industries do our customers work in? The answer may be somewhere in an email, a company description or a website. Yet until someone turns that information into a usable field, a report cannot do much with it.

I encounter this gap in both research and business projects: the information is available (in theory), but not in the desired format. If getting the data in the desired format requires to label a few dozen records, then it is manageable, but doing the same for thousands quickly becomes a project in itself, and is then often discarded altogether. This raises a practical question: how much of that work can we reasonably ask AI to do?

Data annotation is one of the more useful applications I have found for generative AI. It involves adding labels or other information to existing data: a category, a true-or-false value, or a short description. Drawing on three practical examples from my own work, I will illustrate both the possibilities and the choices that determine whether the results are useful.

Research on AI annotation

Before turning to practice, it is worth asking what research says about how well AI annotates, and how to do it systematically. Törnberg (2023) compared GPT-4 with crowd workers and two political science researchers on a specific task: inferring whether a tweet came from a Republican or Democratic politician. His sample contained 500 tweets, equally divided between the parties, whose authors’ party affiliations supplied the reference answers (gold standard). GPT-4 outperformed both expert annotators and the combined crowd-worker answers, while also showing greater consistency.

That is encouraging, although it does not establish that LLMs outperform people across annotation tasks. In an overview of twelve studies, Pavlovic & Poesio (2024) found promising results where evaluation centres on the majority label. Their own experiments also exposed a limitation: models did not reliably reproduce the diversity of human opinions on subjective questions. A single label can be useful whilst still concealing meaningful disagreement.

The practical response is to define the task carefully and test it on the data concerned. Törnberg (2024) proposes a systematic coding procedure that starts with a codebook: an explicit description of the categories and the rules for assigning them. Human annotators and the model label a small development sample, disagreements are examined, and the instructions are refined. Once the prompt is settled, a separate validation sample can provide a measure of performance.

The output format also matters. Using JSON (a structured text format that computers can read easily) with a predefined output schema makes the model’s responses straightforward to validate and process. Finally, the choice of model is frequently relevant to weigh in different factors. Törnberg (2024) names, among other factors, ethical and legal standards, such as: where is the data processed, and is it used for training? Also support for the language of your data, and whether the model is capable (“intelligent”) enough to handle the complexity of the task at hand, are factors to take into account (Törnberg, 2024).

Retrieving relevant case law

For my BSc Thesis (Computer Science, University of Amsterdam), I built a model of the actio pauliana, the provision of the Dutch Bankruptcy Act (Art. 42 Fw) that allows an insolvency practitioner to avoid transactions that prejudiced creditors (Boven, 2026; Boven et al., 2026). Part of that model retrieves relevant previous judgments from rechtspraak.nl. For that retrieval to work well, each judgment first had to be labelled: does the court actually rule on one or more of the three open-textured elements?

For each element, the model returned a true-or-false value. A positive label meant that the court addressed the element, regardless of whether it found that element satisfied. These labels allowed later retrieval to focus on judgments relevant to a particular legal question.

I compared GPT-5.4 and Mistral Medium 3.5 against a manually annotated sample of 100 judgments. The results reveal a useful trade-off:

Table 1. Annotation performance per element against 100 manually annotated judgments.

Precision asks how many positive labels were correct. Recall asks how many of the genuinely positive instances the model found. GPT produced fewer false positives; Mistral missed fewer relevant instances. For this retrieval workflow, Mistral was chosen based on recall, as it can be worse for a lawyer to miss a relevant judgment than having to skim past an irrelevant one. Also, Mistral Medium 3.5 is an open-weight model which means that it can be run with different AI (inference) providers, which benefits reproducibility.

These results support a particular use of annotation within a research workflow. They also show why choosing a model involves more than comparing a single overall score: the errors that matter depend on what happens next.

Adding industries to a customer database

A similar problem arose during a CRM migration for a company in the roofing industry. We wanted to classify its business contacts by industry, using categories agreed with the client. The existing records often contained too little information to do this confidently.

I therefore supplemented the CRM data with website content. With Crawl4AI, an open-source web crawler, I collected the text of the company’s website based on the CRM website field or the business email domains. The model then received the company details, the collected website text and the classification guidelines. Those guidelines also addressed a less obvious problem: a contact person’s email domain can belong to a different organisation. The model had to assess which website matched the company before using its content.

Because this was the client’s own customer data, I used Mistral Medium 3.5 through the French cloud provider Scaleway: the data stayed within the European Union and was not used to train new models. The LLM was prompted to assign each company one industry from the predefined list, with a one-sentence explanation. The resulting dataset covered nearly 2100 companies.

Rather than asking the model how certain it was (LLMs are poor judges of their own confidence), I derived a score from the probabilities the model assigns to each word fragment (token) it generates. I combined the probabilities of the tokens making up the chosen industry label. The model generated a short explanation first, so these probabilities were conditioned on that explanation.

These scores provided an indication of uncertainty, rather than a guarantee of correctness. I used them to prioritise manual review, and an employee checked and corrected some of the lower-scoring classifications.

The practical gain was a database that could support reporting by customer industry. Information previously scattered across records and websites became available for analysing customer groups and informing commercial decisions. The value lies in what those additional fields make possible.

Giving sales leads useful titles

The same migration presented a smaller, rather familiar difficulty. Many sales leads lacked a short, descriptive title, although the new system required one. The relevant information was often already present in customer questions, internal notes and emails.

I used that material to generate brief Dutch titles, generally no more than about eight words, describing the subject and identifying the company or person. Where information was sparse, the model could fall back on the lead’s source and name.

In my experience, the results were useful in most cases. Processing more than 2,700 leads cost less than €4 in AI model usage, excluding development and review time. Strictly speaking, generating titles is a short summarisation task, extending the same approach to data enrichment beyond fixed categories.

Choosing the infrastructure and the model

For the business workflows, I used Mistral through the French cloud provider Scaleway. Its service runs on European infrastructure, does not use customer data to train the underlying models, and applies a zero-retention policy by default, with limited troubleshooting and security exceptions (Scaleway, 2025).

Model choice can also be proportionate to the task. A classifier assigns an input to one of a predefined set of categories (Jurafsky & Martin, 2026, § 4.1). Although general-purpose LLMs can perform this task, specialised models may offer a more efficient approach when the required output is a boolean or an industry label.

One recent example is TypeSafe’s Jev, introduced in September 2026. It returns predefined, typed outputs and associated probabilities, with its developer reporting substantial speed and cost advantages over general-purpose LLMs (Almeida, 2026).

Laya offers an open-weight approach to similar decision tasks, allowing the model to run on infrastructure chosen by the user (Convai Innovations, n.d.). For either approach, the practical question is how much efficiency can be gained while maintaining sufficient accuracy on the intended task.

Conclusion

The public debate in AI sometimes tends to frame automation with generative AI as something to fear, but these examples, amongst others, give me reason to be optimistic about applying AI in data annotation workflows. Human effort remains necessary to establish categories, resolve ambiguity and assess the output. Once that groundwork is in place, AI can make work feasible at a scale that would otherwise be difficult to justify.

As AI shifts from fixed subscriptions to payment per token, cost will matter more, and there lies perhaps the most useful takeaway. This makes it worth asking which model offers sufficient accuracy at an acceptable cost for the task at hand. For the applications outlined above, that turned out to be more modest than it might have seemed at first.

Making that choice requires knowing what a useful result looks like. The strongest results begin with a well-defined question, suitable source material and a clear idea of which mistakes are acceptable.

A database full of correspondence already contains a great deal of knowledge. Sometimes the most useful contribution of AI is to help put that knowledge in a field we can actually use.