# ner correct with prodigy 1.11.8

**URL:** <https://support.prodi.gy/t/ner-correct-with-prodigy-1-11-8/6131>\
**Category:** Uncategorized\
**Tags:** ner\
**Created:** [November 18, 2022, 1:38pm UTC](https://support.prodi.gy/t/ner-correct-with-prodigy-1-11-8/6131 "2022-11-18T13:38:33Z")\
**Posts on this page:** 1\
**Showing post:** 12

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [December 30, 2022, 7:37pm UTC](https://support.prodi.gy/t/ner-correct-with-prodigy-1-11-8/6131/12 "2022-12-30T19:37:09Z")

</div>

hi @JulieSarah!

> [@JulieSarah](#):
>
> Still, I don't understand some things:

So mismatched tokenization can be a big problem that many users don't realize how important it is until it happens. Typically this happens when users have pre-annotated spans/entities that they load into manual recipes. These would be spans/entities they annotated in another tool, formatted the data for Prodigy, and then they use in the manual recipe (e.g., `ner.manual`) with a different tokenizer (e.g., `blank:en` or `en_core_web_sm`).

This is a good post that highlights it and provides some context on how to identify.

> [@Mismatched Tokenization on NER](https://support.prodi.gy/t/mismatched-tokenization-on-ner/4353/2):
>
> Hi! Are you able to find that particular example it fails on in your data? The basic problem here is this: the character offsets in your annotations are referring to spans of texts that do not map to the token boundaries produced by the tokenizer. This means that an NER model, which predicts token-based tags, couldn't be updated with this information and learn anything from it, because it will nevr actually get to see those tokens at runtime. For instance, imagine you have a text like "AB C", w…

> [@JulieSarah](#):
>
> I trained a model with a new tokenizer I created.
> 
> The annotation where the model is trained have been annotated with a default tokenizer.

When you say you had a new tokenizer you created -- was this from scratch like this:

> **[Linguistic Features · spaCy Usage Documentation](https://spacy.io/usage/linguistic-features#native-tokenizers)**
>
> spaCy is a free open-source library for Natural Language Processing in Python. It features NER, POS tagging, dependency parsing, word vectors and more.

Did you train a `ner` component in a spaCy pipeline which included a custom tokenizer that was different than the `ner` annotations used to train?

Yeah - that could cause problems. Is there a compelling reason why you didn't make your annotations with the same custom tokenizer that you intend to include in the pipeline?

I suspect it may be you made the annotations first. Then after reviewing those, you found some problems with the spaCy (default tokenizer) so you decided to build a custom tokenizer but didn't want to redo all of the annotations.

> [@JulieSarah](#):
>
> My goal is to correct past annotation with new tokenizer. Is it feasible?

I would recommend using the code from [the earlier post](https://support.prodi.gy/t/mismatched-tokenization-on-ner/4353/2) to determine which annotations have a mismatch.

```python
nlp = spacy.load("my_custom_model") # model with your custom_tokenizer

for example in examples: # your existing annotations
    doc = nlp(example["text"])
    for span in example["spans"]:
        char_span = doc.char_span(span["start"], span["end"])
        if char_span is None: # start and end don't map to tokens
            print("Misaligned tokens", example["text"], span)

```

Typically, it may not be a lot of examples that have mismatch. You could re-annotate only those annotations that have mismatch but this time with your custom tokenizer.

Alternatively, you could try this package to "align" your annotations' tokenization to your new custom tokenizer:

> **[GitHub - explosion/spacy-alignments: 💫 A spaCy package for Yohei Tamura's...](https://github.com/explosion/spacy-alignments)**
>
> 💫 A spaCy package for Yohei Tamura's Rust tokenizations library - GitHub - explosion/spacy-alignments: 💫 A spaCy package for Yohei Tamura's Rust tokenizations library

I haven't used this package so I can't provide a lot more suggestions (but hopefully the package is self-explanatory).

Hope this helps!

---

_[View the full topic](https://support.prodi.gy/t/ner-correct-with-prodigy-1-11-8/6131)._