# Custom model, ner train, score always 0

**URL:** <https://support.prodi.gy/t/custom-model-ner-train-score-always-0/5488>\
**Category:** Uncategorized\
**Tags:** usage, ner, custom, training\
**Created:** [March 28, 2022, 6:38pm UTC](https://support.prodi.gy/t/custom-model-ner-train-score-always-0/5488 "2022-03-28T18:38:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![linb](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/linb/32/2868_2.png) [@linb](https://support.prodi.gy/u/linb)\
**Post date:** [March 28, 2022, 6:38pm UTC](https://support.prodi.gy/t/custom-model-ner-train-score-always-0/5488/1 "2022-03-28T18:38:09Z")

</div>

I created a custom model, like this

```python
import spacy
from spacy.tokenizer import Tokenizer
nlp = spacy.load("en_core_web_md")

# split prevNameTail, prev_name_tail, obj.child.key
infixes = tuple([r"[A-Z][a-z0-9]+", r"[a-z0-9A-Z]+"]) + tuple(nlp.Defaults.prefixes) + tuple([r"[^\w\s]+"])
infix_re = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer = Tokenizer(nlp.vocab, infix_finditer=infix_re.finditer)
nlp.to_disk('my_en_core_web_md')

```

And train command:

```python
python -m prodigy train my-model --ner gold_ner --base-model my_en_core_web_md --label-stats --eval-split 0.2

```

Tried many times, score always 0:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/b/bcd9cfc983697881c7bc69e6611e1da84fd9a8d2.png)

---

<div class="post-metadata">

**Author:** ![ljvmiranda921](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ljvmiranda921/32/3197_2.png) [@ljvmiranda921](https://support.prodi.gy/u/ljvmiranda921)\
**Post date:** [March 29, 2022, 1:06am UTC](https://support.prodi.gy/t/custom-model-ner-train-score-always-0/5488/2 "2022-03-29T01:06:31Z")

</div>

Hi @linb ,

Based from your screenshot, it seems that you only have 10 examples in total, is this intentional? Usually there's not much to learn from 8 examples, and not much to evaluate with 2 (i.e., if you get both wrong, then the accuracy is 0). In a way, the numbers we see during training makes sense given the number of samples we have.

My suggestion is to try it out in a relatively large sample of data. You can go little by little, first in the order of hundreds and so on.

---

<div class="post-metadata">

**Author:** ![linb](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/linb/32/2868_2.png) [@linb](https://support.prodi.gy/u/linb)\
**Post date:** [March 29, 2022, 12:09pm UTC](https://support.prodi.gy/t/custom-model-ner-train-score-always-0/5488/3 "2022-03-29T12:09:28Z")

</div>

Thanks for your advice. I added 99 examples. Same result:

```python
python -m prodigy train scripts-model-1 --ner script_gold_ner-1 --base-model Arganteal_en_core_web_md --label-stats --eval-split 0.25

```

 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/1/174374b3b7a54296c434acd38756da64bb046f6f.png)

I think , the reason maybe - when use customized '--base-model', the prodigy ignore the new tokenizer.

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [March 30, 2022, 10:50am UTC](https://support.prodi.gy/t/custom-model-ner-train-score-always-0/5488/4 "2022-03-30T10:50:27Z")

</div>

According to the logs, you still only have 24 evaluation examples so it's a bit difficult to draw meaningful conclusions here since the evaluation data is so small.

> [@linb](#):
>
> I think , the reason maybe - when use customized '--base-model', the prodigy ignore the new tokenizer.

If your data requires your custom tokenizer and may not predict accurately if it's not avaliable, this could be something to look into. If your base model is only intended to provide the tokenizer and no trained components to update, can you add it via the `--config` instead and leave out the `--base-model`?

---

<div class="post-metadata">

**Author:** ![linb](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/linb/32/2868_2.png) [@linb](https://support.prodi.gy/u/linb)\
**Post date:** [March 30, 2022, 2:06pm UTC](https://support.prodi.gy/t/custom-model-ner-train-score-always-0/5488/5 "2022-03-30T14:06:38Z")

</div>

If I use --config, how can I modify the tokenizer setting to my custom tokenizer?

```python
[nlp.tokenizer]
@tokenizers = "spacy.Tokenizer.v1" ???

```

Let's say, the tokenizer file is my\_tokenizer.py

```python
import spacy
from spacy.tokenizer import Tokenizer
from spacy.util import compile_infix_regex

@spacy.registry.tokenizers("my_tokenizer")
def create_my_tokenizer():
    infixes = (r".",)
    infix_re = compile_infix_regex(infixes)

    def create_tokenizer(nlp):
        return Tokenizer(
                    nlp.vocab, 
                    suffix_search=None,
                    prefix_search=None,
                    infix_finditer=infix_re.finditer,
                    token_match=None)

    return create_tokenizer

```

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [March 30, 2022, 2:59pm UTC](https://support.prodi.gy/t/custom-model-ner-train-score-always-0/5488/6 "2022-03-30T14:59:07Z")

</div>

Yes, that's pretty much it 🙂 You'd then set the tokenizer to your custom tokenizer:

```python
[nlp.tokenizer]
@tokenizers = "my_tokenizer"

```

... and use the `-F` option in `prodigy train` to point at the file including the code, e.g. `-F my_tokenizer.py`.