How to use Scorer function toevaluate a custom model.

, ,
1

Hi

I am using spacy3.5 for evaluation while my model was generated using spacy3.5.

I have some data from manual annotation which I want to use to evaluate a custom model built in spacy.
I am unable to find the relevant tutorial/documentation about how to go about it .
My code is below.

nlp = spacy.load("model-best")
TEST_DATA = [("The type strain is ST 57T (=ATCC BAA-2401T=DSM 25251T), isolated from the trachea of 
a white stork nestling in Nielitz, Mecklenburg-Western Pomerania, Germany.",{"entities":[(19,25,"strain"), 
(56,69,"iso"),(74,81,"iso_loc"),(87,98,"organism"),(110,118,"location"),(120,149,"location"), (151,158,"location")]}),
         ("Temperature range for growth is 23- 44 C, with optimum growth at 37 C.",{"entities":[(0,11,"temp"), 
(22,28,"growth"),(32,40,"temp"),(47,61,"growth"),(65,69,"temp")]})]
for text, annotations in TEST_DATA:
    doc_pred = nlp(text)
    example = Example.from_dict(doc_pred, {"entities": entity_offsets})
    scores = scorer.score(examples)

Also, I want to compute Precision and Recall with respect to each entity tag.

Thanks

2

hi @aryhalder!

Thanks for your question and welcome to the Prodigy community :wave:

I think you're almost there. Try:

examples = []
for text, annotations in TEST_DATA:
    doc_pred = nlp(text)
    example = Example.from_dict(doc_pred, annotations)
    examples.append(example)
scores = scorer.score(examples) # fyi, use scorer.score_spans(examples, "ents") for only ent scores

Here's a reproducible example:

import spacy
from spacy.scorer import Scorer
from spacy.training import Example

# Default scoring pipeline
scorer = Scorer()

nlp = spacy.load("en_core_web_sm")
TEST_DATA = [
    ("Who is Shaka Khan?", {"entities": [(7, 17, "PERSON")]}),
    ("My name is Ivan", {"entities": [(11, 15, "PERSON")]}),
]

examples = []
for text, annotations in TEST_DATA:
    doc_pred=nlp(text)
    example=Example.from_dict(doc_pred, annotations)
    examples.append(example)
print(scorer.score_spans(examples, "ents"))
# {'ents_p': 0.5, 'ents_r': 0.5, 'ents_f': 0.5, 'ents_per_type': {'PERSON': {'p': 1.0, 'r': 0.5, 'f': 0.6666666666666666}, 'ORG': {'p': 0.0, 'r': 0.0, 'f': 0.0}}}

FYI, this forum is for Prodigy, not spaCy. There's a different forum for spaCy questions on its GitHub discussions forum. I would recommend posting there next time if your question is spaCy-specific. When posting there be sure to read their FAQ first.

For example, there were related posts/issues that may have helped:

Hope this helps!