# How to use Scorer function toevaluate a custom model.

**URL:** <https://support.prodi.gy/t/how-to-use-scorer-function-toevaluate-a-custom-model/6334>\
**Category:** Uncategorized\
**Tags:** usage, spacy, ner\
**Created:** [February 1, 2023, 1:53pm UTC](https://support.prodi.gy/t/how-to-use-scorer-function-toevaluate-a-custom-model/6334 "2023-02-01T13:53:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![aryhalder](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/aryhalder/32/3584_2.png) [@aryhalder](https://support.prodi.gy/u/aryhalder)\
**Post date:** [February 1, 2023, 1:53pm UTC](https://support.prodi.gy/t/how-to-use-scorer-function-toevaluate-a-custom-model/6334/1 "2023-02-01T13:53:53Z")

</div>

Hi

I am using spacy3.5 for evaluation while my model was generated using spacy3.5.

I have some data from manual annotation which I want to use to evaluate a custom model built in spacy.  
I am unable to find the relevant tutorial/documentation about how to go about it .  
My code is below.

```
nlp = spacy.load("model-best")
TEST_DATA = [("The type strain is ST 57T (=ATCC BAA-2401T=DSM 25251T), isolated from the trachea of 
a white stork nestling in Nielitz, Mecklenburg-Western Pomerania, Germany.",{"entities":[(19,25,"strain"), 
(56,69,"iso"),(74,81,"iso_loc"),(87,98,"organism"),(110,118,"location"),(120,149,"location"), (151,158,"location")]}),
         ("Temperature range for growth is 23- 44 C, with optimum growth at 37 C.",{"entities":[(0,11,"temp"), 
(22,28,"growth"),(32,40,"temp"),(47,61,"growth"),(65,69,"temp")]})]
for text, annotations in TEST_DATA:
    doc_pred = nlp(text)
    example = Example.from_dict(doc_pred, {"entities": entity_offsets})
    scores = scorer.score(examples)

```

Also, I want to compute Precision and Recall with respect to each entity tag.

Thanks

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [February 1, 2023, 8:18pm UTC](https://support.prodi.gy/t/how-to-use-scorer-function-toevaluate-a-custom-model/6334/2 "2023-02-01T20:18:11Z")

</div>

hi @aryhalder!

Thanks for your question and welcome to the Prodigy community 👋

I think you're almost there. Try:

```python
examples = []
for text, annotations in TEST_DATA:
    doc_pred = nlp(text)
    example = Example.from_dict(doc_pred, annotations)
    examples.append(example)
scores = scorer.score(examples) # fyi, use scorer.score_spans(examples, "ents") for only ent scores

```

Here's a reproducible example:

```python
import spacy
from spacy.scorer import Scorer
from spacy.training import Example

# Default scoring pipeline
scorer = Scorer()

nlp = spacy.load("en_core_web_sm")
TEST_DATA = [
    ("Who is Shaka Khan?", {"entities": [(7, 17, "PERSON")]}),
    ("My name is Ivan", {"entities": [(11, 15, "PERSON")]}),
]

examples = []
for text, annotations in TEST_DATA:
    doc_pred=nlp(text)
    example=Example.from_dict(doc_pred, annotations)
    examples.append(example)
print(scorer.score_spans(examples, "ents"))
# {'ents_p': 0.5, 'ents_r': 0.5, 'ents_f': 0.5, 'ents_per_type': {'PERSON': {'p': 1.0, 'r': 0.5, 'f': 0.6666666666666666}, 'ORG': {'p': 0.0, 'r': 0.0, 'f': 0.0}}}

```

FYI, this forum is for Prodigy, not spaCy. There's a different forum for spaCy questions on its [GitHub discussions forum](https://github.com/explosion/spacy/discussions). I would recommend posting there next time if your question is spaCy-specific. When posting there be sure to read their [FAQ](https://github.com/explosion/spaCy/discussions/8226) first.

For example, there were related posts/issues that may have helped:

> **[Spacy v3 example object for scorer function · Discussion #8897 · explosion/spaCy](https://github.com/explosion/spaCy/discussions/8897)**
>
> Hi everyone, I wanted to run the scorer class to evaluate my custom NER trained model on a validation dataset. the dataset looks like this: Valid\_data= \[ "Mr. and Mrs. Dursley, of number four,...

Hope this helps!