Mismatched tokenization

[ryanwesslen]

hi @ttandel!

It seems the character offsets in your annotations are referring to spans of texts that do not map to the token boundaries produced by the tokenizer (blank:en) you're using in ner.manual.

What tokenizer did you use to get the pre-set spans?

Any chance you could redo the pre-set spans to use the same tokenizer as ner.manual (i.e., use the blank:end)?

This would be the easiest fix but I suspect it may not be possible. There are a lot (~39) posts with the keyword "mismatched tokenization" that can help.

I couldn't go through all of them, but this one gives you a snippet of code to identify which are the mismatched spans:

Mismatched Tokenization on NER

Hi! Are you able to find that particular example it fails on in your data? The basic problem here is this: the character offsets in your annotations are referring to spans of texts that do not map to the token boundaries produced by the tokenizer. This means that an NER model, which predicts token-based tags, couldn't be updated with this information and learn anything from it, because it will nevr actually get to see those tokens at runtime. For instance, imagine you have a text like "AB C", w…

Similarly, as this post discusses, the best course of action depends on what are the mismatches:

Matching tokenisation on pre-existing annotated data

Hi! The best approach depends on what the underlying mismatches are. If you need a better overview, you could write a small script that processes your texts with spaCy's default tokenizer and then calls doc.char_span with the start and end offsets. If that returns None, you have a mismatch. Maybe you'll find that it mostly comes down to things like punctuation (e.g. hyphens) that can be adjusted by slightly tweaking spaCy's tokenization to better match your data. Or maybe it turns out that some…

See if you can use these posts to help and feel free to post back questions if you run into issues (or let us know if you're able to fix it!)

Also, a small ask, moving forward, please refrain from posting images of the code and/or output. It's easier to use the markdown code output as this will allow us to copy/paste (especially for code) and also make the code searchable instead of images. Thank you