Guide · Annotation guidelines
Entity extraction that follows your annotation guidelines
Updated 19 November 2026, by Jungroo AI Labs, the makers of ReDefiNER
Annotation guidelines say what counts as each entity type: a definition, rules for the edge cases and a few examples. Guideline-following models read them when they run, so changing what counts means editing the guidelines, not retraining. GoLLIE showed the idea with 7B to 34B models; ReDefiNER does it with 0.8B parameters on a CPU. Here is how it does on guidelines written by others, where it loses, and which rules it follows.
What a guideline looks like to ReDefiNER
A rulebook: general instructions, then each type with a one-line description and numbered rules. The same model gives different answers when you change the text:
[RULES] Extract every entity of the supplied types. Return exact spans from the text. [TYPE] ORDER_ID [DESCRIPTION] A customer's order reference. OR1 A reference starting with # is an order number: "#D-55102". [TYPE] INVOICE_ID [DESCRIPTION] An invoice reference. IN1 A reference starting with INV- is an invoice number: "INV-7731". [TYPE] COMPANY [DESCRIPTION] A company or brand. CO1 Our own company and products, "Acme" and "Acme Home", are never labelled.
Existing guidelines convert directly: for the results below, GoLLIE's guideline text and five examples per type became a rulebook with nothing added. How to write rules that work.
With GoLLIE's own guidelines
GoLLIE's 13 NER test sets, which its authors use to test unseen tasks, scored with GoLLIE's code. ReDefiNER's scores are the mean over 3 orders of the types in the rulebook; GoLLIE's are from its paper.
| Dataset | ReDefiNER 0.8B | GoLLIE 7B | GoLLIE 13B | GoLLIE 34B |
|---|---|---|---|---|
| FabNER | 32.7 | 24.8 | 25.4 | 26.3 |
| MultiNERD | 82.0 | 76.0 | 77.5 | 77.2 |
| BroadTwitter | 45.5 | 49.5 | 51.4 | 50.3 |
| HarveyNER | 33.5 | 37.3 | 41.3 | 38.9 |
| E3C | 47.7 | 59.0 | 59.0 | 60.0 |
| WikiEvents | 46.4 | 80.7 | 80.2 | 81.3 |
| CrossNER AI | 66.7 | 59.1 | 56.7 | 61.6 |
| CrossNER Literature | 70.1 | 62.7 | 59.7 | 59.1 |
| CrossNER Music | 73.7 | 67.8 | 65.5 | 68.4 |
| CrossNER Politics | 78.0 | 57.2 | 54.4 | 60.2 |
| CrossNER Science | 66.0 | 55.5 | 56.2 | 56.3 |
| MIT Movie | 71.3 | 63.0 | 62.5 | 62.4 |
| MIT Restaurant | 63.8 | 43.4 | 49.8 | 52.7 |
| Mean, all 13 | 59.8 | 56.6 | 56.9 | 58.1 |
ReDefiNER is ahead of GoLLIE 34B on 9 of the 13 sets with about 1/40 of its parameters. It loses on BroadTwitter, HarveyNER, E3C and, by the most, WikiEvents, which labels pronouns and common nouns under the ACE05 scheme GoLLIE was trained on.
When the guidelines are rewritten
- Rules tuned on dev data: 67.9. Adding our own rules after looking at dev-set errors lifts the 13-set mean from 59.8 to 67.9. GoLLIE didn't get the same help, so this is not a like-for-like comparison.
- BUSTER, a loss. On this financial-documents set, SLIMER 7B reports 45.3 micro-F1. ReDefiNER scores 22.1 with SLIMER's type wording and 35.8 with rules tuned on dev.
- Other guideline-following models. On the 7 CrossNER and MIT sets, ReDefiNER averages 69.9 with GoLLIE's guidelines. SLIMER 7B, given definitions and guidelines, reports 54.0 and KnowCoder 7B 60.1. B2NER 20B reports 72.1 and B2NER 7B 69.8, given type names only.
Against frontier models with the same guidelines
Given GoLLIE's CrossNER AI guidelines as the rulebook, five frontier models scored 72.4 to 79.4 F1 and ReDefiNER 68.1. ReDefiNER costs about $0.02 per 1,000 requests on a 2-vCPU server, against $0.19 for Claude Haiku 5.5 and $2.42 for Claude Opus 5.5, and keeps the text on your servers. Method and full table.
Which rules it follows, and which it ignores
- Followed: exclusions (“leave out titles”), never-label lists (“our own brand”), format rules (“a reference starting with # is an order number”), and precedence rules written as numbered rules with worked examples.
- Followed with care: pronouns, when they are quoted in a short list under the type.
- Mostly ignored: rules against a strong prior, such as relabelling a person's name as something else, asking for common nouns as entities, or “only revenue amounts, not every money amount”.
- Mostly ignored: rules that need counting or judging what didn't happen.
- Weak: abstaining. When a type's condition isn't clearly met, it tends to label anyway.