# Matching tokenisation on pre-existing annotated data

**URL:** <https://support.prodi.gy/t/matching-tokenisation-on-pre-existing-annotated-data/2710>\
**Category:** Uncategorized\
**Tags:** usage, ner, spacy, solved\
**Created:** [March 27, 2020, 7:43am UTC](https://support.prodi.gy/t/matching-tokenisation-on-pre-existing-annotated-data/2710 "2020-03-27T07:43:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![barnybug](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/barnybug/32/1286_2.png) [@barnybug](https://support.prodi.gy/u/barnybug)\
**Post date:** [March 27, 2020, 7:43am UTC](https://support.prodi.gy/t/matching-tokenisation-on-pre-existing-annotated-data/2710/1 "2020-03-27T07:43:53Z")

</div>

We have pre-existing data annotated for NER that I'd like to use prodigy to review and correct before training an NER model on it. It's in the (text, {entitiies: ...}) spacy training format but not tokenised.

Having converted this to {text: ..., spans: ...} as input to ner.manual, a few examples load fine but some of the existing spans do not exactly land on the tokens the model is generating and I see a 'ValueError: Mismatched tokenization'.

We'd like to 'snap' the existing spans to agree with spacy tokenisation, so my question is how's the best way to go about this?

thanks in advance!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [March 27, 2020, 10:51am UTC](https://support.prodi.gy/t/matching-tokenisation-on-pre-existing-annotated-data/2710/2 "2020-03-27T10:51:28Z")

</div>

Hi! The best approach depends on what the underlying mismatches are. If you need a better overview, you could write a small script that processes your texts with spaCy's default tokenizer and then calls [`doc.char_span`](https://spacy.io/api/doc#char_span) with the start and end offsets. If that returns `None`, you have a mismatch.

Maybe you'll find that it mostly comes down to things like punctuation (e.g. hyphens) that can be adjusted by slightly [tweaking spaCy's tokenization](https://spacy.io/usage/linguistic-features#native-tokenizer-additions) to better match your data. Or maybe it turns out that some of your existing annotated entities have off-by-one errors or include leading or trailing whitespace. That can be easily fixed programmatically by adjusting the start / end offset by `1` if the text span starts or ends with whitespace.

If you find cases with actual mistakes that need to be re-annotated, you could also queue them up again for annotation in Prodigy without the mismatched `"spans"` (and maybe add the info about the original entity to the task's `"meta"` so it's displayed in the corner and you can try to reannotate it using the existing tokenization).

---

<div class="post-metadata">

**Author:** ![barnybug](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/barnybug/32/1286_2.png) [@barnybug](https://support.prodi.gy/u/barnybug)\
**Post date:** [March 27, 2020, 3:46pm UTC](https://support.prodi.gy/t/matching-tokenisation-on-pre-existing-annotated-data/2710/3 "2020-03-27T15:46:27Z")

</div>

Thanks for the swift reply. The pointer to `char_span` was very helpful. Most were slight whitespace disagreement and readily fixed.