Hi @angelo!
Apologies for the delay. Yes, it is defnitely possible to preannotate data with coreference labels using Prodigy and spacy-llm. Let me address each point separately:
Is it possible to use an LLM to help annotate coreference examples for Spanish?
Yes, coreference annotation usually involves two steps:
- NER / entity detection
- coreference relations between the detected entities
LLM can defnitely do both, but it's recommended to these steps separately because:
- if the LLM gets a mention boundary wrong, the coreference link built on top of it is also wrong. Separating the steps means you can fix mention detection without redoing the linking.
- Annotation efficiency: annotators who curate LLM's annotations focus on one task at a time, which is known to be less error prone.
- Prompt reliability: a prompt that asks "given these numbered mentions, which pairs corefer?" is far more constrained than "find all mentions and group them", and produces more consistent LLM output.
How could I create a prompt or template to support this task?
The cleanest approach uses spacy-llm built-in spacy.REL.v1 task, which handles prompt generation and response parsing automatically. You configure the prompt in a spaCy .cfg file and write a small Prodigy recipe to wire it up i.e. translate to spacy-llm output to the Prodigy task dictionary structure than can be used by relations UI.
Here's an example .cfg file. All LLM configuration lives here. You can swap model or tune the prompt without touching Python:
[nlp]
lang = "es"
pipeline = ["llm"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 256
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
[components]
[components.llm]
factory = "llm"
save_io = false
[components.llm.task]
@llm_tasks = "spacy.REL.v1"
labels = ["COREF"]
[components.llm.task.label_definitions]
COREF = "Dos menciones se refieren a la misma entidad del mundo real. Ejemplo: 'María' y 'Ella' son correferentes porque se
refieren a la misma persona."
[components.llm.model]
@llm_models = "spacy.GPT-4.v3"
[components.llm.model.config]
temperature = 0
Then you'd need to write a custom Prodigy recipe that translates the spacy-llm annotations to Prodigy task dictionary compatible with the relations UI.
You can revise how built-in spacy-llm recipes for NER & textcat work, but it could look something like this:
import spacy_llm
from spacy_llm.util import assemble
from prodigy.core import recipe, Arg
from prodigy.components.stream import get_stream
def _tokens_from_doc(doc):
return [
{"text": tok.text, "start": tok.idx, "end": tok.idx + len(tok.text),
"id": tok.i, "ws": bool(tok.whitespace_)}
for tok in doc
]
def add_spacy_llm_coref_relations(stream, *, nlp):
for eg in stream:
text = eg.get("text", "")
input_spans = eg.get("spans", [])
doc = nlp.make_doc(text)
# Extract tokens before running the LLM — spacy-llm injects entity
# markers into the text for prompting, which corrupts the token list
# if extracted afterwards.
tokens = _tokens_from_doc(doc)
# Set doc.ents from the pre-annotated input spans so the REL task
# knows which mentions to consider for linking.
ents, resolved_spans = [], []
for s in input_spans:
span = doc.char_span(s["start"], s["end"], label=s["label"],
alignment_mode="expand")
if span is None:
print(f"[warn] could not align '{s['text']}' to tokens — skipping")
continue
ents.append(span)
resolved_spans.append({**s, "token_start": span.start, "token_end": span.end - 1})
try:
doc.ents = ents
doc = nlp(doc) # spacy-llm prompts the LLM and parses the response
except Exception as exc:
print(f"[LLM error] {exc}")
eg["tokens"] = tokens
eg["spans"] = resolved_spans
eg["relations"] = []
yield eg
continue
# Convert spacy-llm RelationItem(dep, dest, relation) to Prodigy format.
# Normalize direction so child is always the anaphor (later in text)
# and head is always the antecedent (earlier in text).
relations = []
for rel in doc._.rel:
if rel.dep >= len(ents) or rel.dest >= len(ents):
continue
tok_a, tok_b = ents[rel.dep].start, ents[rel.dest].start
relations.append({
"child": max(tok_a, tok_b),
"head": min(tok_a, tok_b),
"label": rel.relation,
})
eg["tokens"] = tokens
eg["spans"] = resolved_spans
eg["relations"] = relations
yield eg
@recipe(
"coref.llm.correct",
dataset=Arg(help="Dataset to save annotations to"),
source=Arg(help="JSONL file with pre-annotated MENTION spans"),
spacy_config=Arg("--config", "-c", help="Path to spacy-llm config file"),
)
def coref_llm_correct(dataset: str, source: str, spacy_config: str = "coref_es.cfg"):
"""Correct LLM-predicted Spanish coreference relations in Prodigy."""
nlp = assemble(spacy_config)
stream = get_stream(source, loader="jsonl", rehash=True, dedup=True)
stream.apply(add_spacy_llm_coref_relations, nlp=nlp)
return {
"view_id": "relations",
"dataset": dataset,
"stream": stream,
"config": {
"labels": ["COREF"],
"relations_span_labels": ["MENTION"],
"wrap_relations": True,
"exclude_by": "input",
},
}
The annotator sees the pre-identified mentions with LLM-predicted coreference arcs already drawn between them, and can accept, correct, or add arcs.
- Could the same approach be adapted to create templates for several other annotation tasks?
Yes — spacy-llm ships built-in tasks for the most common annotation types, all configurable through the .cfg file. You can see the details of the supportesd tasks here. Naturally, you can develop custom tasks as well!
Prodigy also supports several tasks out-of-the-box: notably NER, textcat and spancat. See detailsl here.
- Is there any official guide or recommended documentation for creating custom Prodigy recipes to develop my own annotation workflows?
Plese see out docs on custom recipes and custom interfaces
Let us know if you need any further guidance.
