Guide · Annotation guidelines

Entity extraction that follows your annotation guidelines

Updated 19 November 2026, by Jungroo AI Labs, the makers of ReDefiNER

Annotation guidelines say what counts as each entity type: a definition, rules for the edge cases and a few examples. Guideline-following models read them when they run, so changing what counts means editing the guidelines, not retraining. GoLLIE showed the idea with 7B to 34B models; ReDefiNER does it with 0.8B parameters on a CPU. Here is how it does on guidelines written by others, where it loses, and which rules it follows.

What a guideline looks like to ReDefiNER

A rulebook: general instructions, then each type with a one-line description and numbered rules. The same model gives different answers when you change the text:

[RULES]
Extract every entity of the supplied types. Return exact spans from the text.

[TYPE] ORDER_ID
[DESCRIPTION] A customer's order reference.
OR1 A reference starting with # is an order number: "#D-55102".

[TYPE] INVOICE_ID
[DESCRIPTION] An invoice reference.
IN1 A reference starting with INV- is an invoice number: "INV-7731".

[TYPE] COMPANY
[DESCRIPTION] A company or brand.
CO1 Our own company and products, "Acme" and "Acme Home", are never labelled.

Existing guidelines convert directly: for the results below, GoLLIE's guideline text and five examples per type became a rulebook with nothing added. How to write rules that work.

With GoLLIE's own guidelines

GoLLIE's 13 NER test sets, which its authors use to test unseen tasks, scored with GoLLIE's code. ReDefiNER's scores are the mean over 3 orders of the types in the rulebook; GoLLIE's are from its paper.

DatasetReDefiNER 0.8BGoLLIE 7BGoLLIE 13BGoLLIE 34B
FabNER32.724.825.426.3
MultiNERD82.076.077.577.2
BroadTwitter45.549.551.450.3
HarveyNER33.537.341.338.9
E3C47.759.059.060.0
WikiEvents46.480.780.281.3
CrossNER AI66.759.156.761.6
CrossNER Literature70.162.759.759.1
CrossNER Music73.767.865.568.4
CrossNER Politics78.057.254.460.2
CrossNER Science66.055.556.256.3
MIT Movie71.363.062.562.4
MIT Restaurant63.843.449.852.7
Mean, all 1359.856.656.958.1

ReDefiNER is ahead of GoLLIE 34B on 9 of the 13 sets with about 1/40 of its parameters. It loses on BroadTwitter, HarveyNER, E3C and, by the most, WikiEvents, which labels pronouns and common nouns under the ACE05 scheme GoLLIE was trained on.

When the guidelines are rewritten

  • Rules tuned on dev data: 67.9. Adding our own rules after looking at dev-set errors lifts the 13-set mean from 59.8 to 67.9. GoLLIE didn't get the same help, so this is not a like-for-like comparison.
  • BUSTER, a loss. On this financial-documents set, SLIMER 7B reports 45.3 micro-F1. ReDefiNER scores 22.1 with SLIMER's type wording and 35.8 with rules tuned on dev.
  • Other guideline-following models. On the 7 CrossNER and MIT sets, ReDefiNER averages 69.9 with GoLLIE's guidelines. SLIMER 7B, given definitions and guidelines, reports 54.0 and KnowCoder 7B 60.1. B2NER 20B reports 72.1 and B2NER 7B 69.8, given type names only.

Against frontier models with the same guidelines

Given GoLLIE's CrossNER AI guidelines as the rulebook, five frontier models scored 72.4 to 79.4 F1 and ReDefiNER 68.1. ReDefiNER costs about $0.02 per 1,000 requests on a 2-vCPU server, against $0.19 for Claude Haiku 5.5 and $2.42 for Claude Opus 5.5, and keeps the text on your servers. Method and full table.

Which rules it follows, and which it ignores

  • Followed: exclusions (“leave out titles”), never-label lists (“our own brand”), format rules (“a reference starting with # is an order number”), and precedence rules written as numbered rules with worked examples.
  • Followed with care: pronouns, when they are quoted in a short list under the type.
  • Mostly ignored: rules against a strong prior, such as relabelling a person's name as something else, asking for common nouns as entities, or “only revenue amounts, not every money amount”.
  • Mostly ignored: rules that need counting or judging what didn't happen.
  • Weak: abstaining. When a type's condition isn't clearly met, it tends to label anyway.

Questions

Frequently asked.

What is guideline-following NER?

Named entity recognition where the model reads annotation guidelines (definitions, rules and examples for each type) when it runs, instead of learning a fixed set of types in training. GoLLIE, SLIMER, KnowCoder and ReDefiNER work this way, and so do frontier LLMs given a prompt.

Is ReDefiNER better than GoLLIE?

On GoLLIE's own 13 NER sets and guidelines, ReDefiNER averages 59.8 F1 against 58.1 for GoLLIE 34B and is ahead on 9 of 13 sets, with about 1/40 of the parameters. GoLLIE is far ahead on WikiEvents and E3C.

Can I use my existing annotation guidelines?

Yes. Write each type as [TYPE] with a one-line [DESCRIPTION], then the guideline's rules as numbered lines and a few examples. GoLLIE's guidelines were converted this way with nothing added.

Does it need a GPU?

No. ReDefiNER runs on an ordinary CPU with about 2 to 2.6 GB of memory. On an NVIDIA GPU, with the rulebook loaded, it answers in about 0.05 seconds per request.

Try it on your own text

Test ReDefiNER on 20 of your sentences.

The live demo runs on a CPU. Edit a rule and see the output change, or write to us and we will test a rulebook on your data.

Contact

Tell us what you need.

We partner with a limited number of enterprises at a time, so every engagement gets senior attention. Tell us what you want to build, and our leadership will reply.

We use these details only to reply to you. What you type is saved as you go, so we can still reply if you leave before pressing Send. Privacy

General enquiries

Jungroo AI Labs

Partnerships, projects and products

Leadership

Sethuraman T A

Chief Executive Officer

Engineering

Cibe Hariharan

Chief Technology Officer