# Insert Exception to skip cases where tokens are misaligned.

**URL:** <https://support.prodi.gy/t/insert-exception-to-skip-cases-where-tokens-are-misaligned/3505>\
**Category:** Uncategorized\
**Tags:** usage, ner, spacy\
**Created:** [October 9, 2020, 8:51am UTC](https://support.prodi.gy/t/insert-exception-to-skip-cases-where-tokens-are-misaligned/3505 "2020-10-09T08:51:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![JBunr](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@JBunr](https://support.prodi.gy/u/JBunr)\
**Post date:** [October 9, 2020, 8:51am UTC](https://support.prodi.gy/t/insert-exception-to-skip-cases-where-tokens-are-misaligned/3505/1 "2020-10-09T08:51:59Z")

</div>

I am loading pre-annotated data into `prodigy ner.train`, as i want to reduce annotation workload. Unfortunately, the software used to annotate named entities gives mismatched tokens that do not align with the spacy tokenizer I am using.

I get the following error, and prodigy does not load any more data into the UI. Is there any way I can throw an exception instead, that ensures cases with misaligned entities are ignored by prodigy in the annotation step?

```python
ValueError: Mismatched tokenization. Can't resolve span to token index 594. This can happen if your data contains pre-set spans. Make sure that the spans match spaCy's tokenization or add a 'tokens' property to your task.

```

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 12, 2020, 8:07am UTC](https://support.prodi.gy/t/insert-exception-to-skip-cases-where-tokens-are-misaligned/3505/2 "2020-10-12T08:07:05Z")

</div>

Hi! The easiest solution would be to adjust the recipe script you're using and to set `skip=True` on the [`add_tokens`](https://prodi.gy/docs/api-components#add_tokens) preprocessor. This will not raise the error and instead just skip the span that can't be mapped to tokens.

You can also easily check which spans don't align by using spaCy's [`Doc.char_span`](https://spacy.io/api/doc#char_span) method, which tries to generate a character-based span and returns `None` if it's not possible. This can sometimes be helpful because maybe there's a common pattern/problem that's easy to adjust programmatically (whitespace, offsets etc.). Here's a simple example:

```python
nlp = spacy.blank("en") # whichever model/tokenizer you want to use
for eg in your_examples_here:
    doc = nlp(eg["text"])
    for span in eg.get("spans", []):
        char_span = doc.char_span(span["start"], span["end"])
        if char_span is None:
            print("Misaligned span", span)

```