# Mismatched Tokenization on NER **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner\ **Created:** [June 23, 2021, 2:50pm UTC](https://support.prodi.gy/t/mismatched-tokenization-on-ner/4353 "2021-06-23T14:50:33Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![lnatprodigy](https://avatars.discourse-cdn.com/v4/letter/l/6f9a4e/32.png) [@lnatprodigy](https://support.prodi.gy/u/lnatprodigy)\ **Post date:** [June 23, 2021, 2:50pm UTC](https://support.prodi.gy/t/mismatched-tokenization-on-ner/4353/1 "2021-06-23T14:50:33Z")
Hi, I just got my prodigy license but sadly I'm already stuck. When I try to import my dataset, with already annotated entities, I get errors like this one: > Task exception was never retrieved > future: \ exception=ValueError("Mismatched tokenization. Can't resolve span to token index 109. This can happen if your data contains pre-set spans. Make sure that the spans match spaCy's tokenization or add a 'tokens' property to your task.\n\n{'start': 93, 'end': 109, 'label': 'ARTIKEL', 'token\_start': 13}")\> > Traceback (most recent call last): > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\uvicorn\protocols\http\h11\_impl.py", line 391, in run\_asgi > self.logger.error(msg, exc\_info=exc) > File "C:\Users\szabop\AppData\Local\Programs\Python\Python37\Lib\logging\__init_\_.py", line 1412, in error > self._log(ERROR, msg, args, \*\*kwargs) > File "C:\Users\szabop\AppData\Local\Programs\Python\Python37\Lib\logging\_init_.py", line 1519, in _log > self.handle(record) > File "C:\Users\szabop\AppData\Local\Programs\Python\Python37\Lib\logging\_init_.py", line 1528, in handle > if (not self.disabled) and self.filter(record): > File "C:\Users\szabop\AppData\Local\Programs\Python\Python37\Lib\logging\__init_\_.py", line 762, in filter > result = f.filter(record) > File "cython\_src\prodigy\util.pyx", line 121, in prodigy.util.ServerErrorFilter.filter > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\uvicorn\protocols\http\h11\_impl.py", line 388, in run\_asgi > result = await app(self.scope, self.receive, self.send) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\uvicorn\middleware\proxy\_headers.py", line 45, in **call** > return await self.app(scope, receive, send) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\fastapi\applications.py", line 140, in **call** > await super(). **call** (scope, receive, send) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\applications.py", line 134, in **call** > await self.error\_middleware(scope, receive, send) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\middleware\errors.py", line 178, in **call** > raise exc from None > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\middleware\errors.py", line 156, in **call** > await self.app(scope, receive, \_send) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\middleware\cors.py", line 84, in **call** > await self.simple\_response(scope, receive, send, request\_headers=headers) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\middleware\cors.py", line 140, in simple\_response > await self.app(scope, receive, send) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\middleware\base.py", line 25, in **call** > response = await self.dispatch\_func(request, self.call\_next) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\prodigy\app.py", line 198, in reset\_db\_middleware > response = await call\_next(request) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\middleware\base.py", line 45, in call\_next > task.result() > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\middleware\base.py", line 38, in coro > await self.app(scope, receive, send) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\exceptions.py", line 73, in **call** > raise exc from None > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\exceptions.py", line 62, in **call** > await self.app(scope, receive, sender) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\routing.py", line 590, in **call** > await route(scope, receive, send) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\routing.py", line 208, in **call** > await self.app(scope, receive, send) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\routing.py", line 41, in app > response = await func(request) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\fastapi\routing.py", line 129, in app > raw\_response = await run\_in\_threadpool(dependant.call, \*\*values) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\starlette\concurrency.py", line 25, in run\_in\_threadpool > return await loop.run\_in\_executor(None, func, \*args) > File "C:\Users\szabop\AppData\Local\Programs\Python\Python37\Lib\concurrent\futures\thread.py", line 57, in run > result = self.fn(\*self.args, \*\*self.kwargs) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\prodigy\app.py", line 420, in get\_session\_questions > return \_shared\_get\_questions(req.session\_id, excludes=req.excludes) > File "C:\Users\szabop.virtualenvs\prodigy-2mgOBuS3\lib\site-packages\prodigy\app.py", line 391, in \_shared\_get\_questions > tasks = controller.get\_questions(session\_id=session\_id, excludes=excludes) > File "cython\_src\prodigy\core.pyx", line 223, in prodigy.core.Controller.get\_questions > File "cython\_src\prodigy\core.pyx", line 227, in prodigy.core.Controller.get\_questions > File "cython\_src\prodigy\components\feeds.pyx", line 99, in prodigy.components.feeds.SharedFeed.get\_questions > File "cython\_src\prodigy\components\feeds.pyx", line 111, in prodigy.components.feeds.SharedFeed.get\_next\_batch > File "cython\_src\prodigy\components\preprocess.pyx", line 130, in add\_tokens > File "cython\_src\prodigy\components\preprocess.pyx", line 222, in prodigy.components.preprocess.\_add\_tokens > File "cython\_src\prodigy\components\preprocess.pyx", line 207, in prodigy.components.preprocess.sync\_spans\_to\_tokens > ValueError: Mismatched tokenization. Can't resolve span to token index 109. This can happen if your data contains pre-set spans. Make sure that the spans match spaCy's tokenization or add a 'tokens' property to your task. > > {'start': 93, 'end': 109, 'label': 'ARTIKEL', 'token\_start': 13} I read through all the threads but I haven't really found a solution to this. Here's an example json: ```python [{ "text": "Arbeitnehmer (auch geringfügig Beschäftigte, Lehrlinge), die ihrem Arbeitgeber ein COVID-19-Risiko-Attest vorlegen, haben Anspruch auf Freistellung von der Arbeitsleistung und Fortzahlung des Entgelts, sofern der Arbeitnehmer die Arbeitsleistung nicht im Home-Office erbringen kann oder die Arbeitsbedingungen in der Arbeitsstätte nicht durch geeignete Maßnahmen so gestaltet werden können, dass eine Ansteckung mit COVID-19 mit größtmöglicher Sicherheit ausgeschlossen ist. Mit BGBl II 2020/609 wurde nun der Zeitraum, in dem Freistellungen von Arbeitnehmern mit einem COVID-19-Risiko-Attest möglich sind, bis zum Ablauf des 31. 3. 2021 verlängert.", "spans": [{ "start": 479, "end": 495, "label": "BGBL" }], "tokens": [ { "text": "Arbeitnehmer", "start": 0, "end": 12, "id": 0 }, { "text": "(", "start": 13, "end": 14, "id": 1 }, { "text": "auch", "start": 14, "end": 18, "id": 2 }, { "text": "geringfügig", "start": 19, "end": 30, "id": 3 }, { "text": "Beschäftigte", "start": 31, "end": 43, "id": 4 }, { "text": ",", "start": 43, "end": 44, "id": 5 }, { "text": "Lehrlinge", "start": 45, "end": 54, "id": 6 }, { "text": ")", "start": 54, "end": 55, "id": 7 }, { "text": ",", "start": 55, "end": 56, "id": 8 }, { "text": "die", "start": 57, "end": 60, "id": 9 }, { "text": "ihrem", "start": 61, "end": 66, "id": 10 }, { "text": "Arbeitgeber", "start": 67, "end": 78, "id": 11 }, { "text": "ein", "start": 79, "end": 82, "id": 12 }, { "text": "COVID-19-Risiko-Attest", "start": 83, "end": 105, "id": 13 }, { "text": "vorlegen", "start": 106, "end": 114, "id": 14 }, { "text": ",", "start": 114, "end": 115, "id": 15 }, { "text": "haben", "start": 116, "end": 121, "id": 16 }, { "text": "Anspruch", "start": 122, "end": 130, "id": 17 }, { "text": "auf", "start": 131, "end": 134, "id": 18 }, { "text": "Freistellung", "start": 135, "end": 147, "id": 19 }, { "text": "von", "start": 148, "end": 151, "id": 20 }, { "text": "der", "start": 152, "end": 155, "id": 21 }, { "text": "Arbeitsleistung", "start": 156, "end": 171, "id": 22 }, { "text": "und", "start": 172, "end": 175, "id": 23 }, { "text": "Fortzahlung", "start": 176, "end": 187, "id": 24 }, { "text": "des", "start": 188, "end": 191, "id": 25 }, { "text": "Entgelts", "start": 192, "end": 200, "id": 26 }, { "text": ",", "start": 200, "end": 201, "id": 27 }, { "text": "sofern", "start": 202, "end": 208, "id": 28 }, { "text": "der", "start": 209, "end": 212, "id": 29 }, { "text": "Arbeitnehmer", "start": 213, "end": 225, "id": 30 }, { "text": "die", "start": 226, "end": 229, "id": 31 }, { "text": "Arbeitsleistung", "start": 230, "end": 245, "id": 32 }, { "text": "nicht", "start": 246, "end": 251, "id": 33 }, { "text": "im", "start": 252, "end": 254, "id": 34 }, { "text": "Home-Office", "start": 255, "end": 266, "id": 35 }, { "text": "erbringen", "start": 267, "end": 276, "id": 36 }, { "text": "kann", "start": 277, "end": 281, "id": 37 }, { "text": "oder", "start": 282, "end": 286, "id": 38 }, { "text": "die", "start": 287, "end": 290, "id": 39 }, { "text": "Arbeitsbedingungen", "start": 291, "end": 309, "id": 40 }, { "text": "in", "start": 310, "end": 312, "id": 41 }, { "text": "der", "start": 313, "end": 316, "id": 42 }, { "text": "Arbeitsstätte", "start": 317, "end": 330, "id": 43 }, { "text": "nicht", "start": 331, "end": 336, "id": 44 }, { "text": "durch", "start": 337, "end": 342, "id": 45 }, { "text": "geeignete", "start": 343, "end": 352, "id": 46 }, { "text": "Maßnahmen", "start": 353, "end": 362, "id": 47 }, { "text": "so", "start": 363, "end": 365, "id": 48 }, { "text": "gestaltet", "start": 366, "end": 375, "id": 49 }, { "text": "werden", "start": 376, "end": 382, "id": 50 }, { "text": "können", "start": 383, "end": 389, "id": 51 }, { "text": ",", "start": 389, "end": 390, "id": 52 }, { "text": "dass", "start": 391, "end": 395, "id": 53 }, { "text": "eine", "start": 396, "end": 400, "id": 54 }, { "text": "Ansteckung", "start": 401, "end": 411, "id": 55 }, { "text": "mit", "start": 412, "end": 415, "id": 56 }, { "text": "COVID-19", "start": 416, "end": 424, "id": 57 }, { "text": "mit", "start": 425, "end": 428, "id": 58 }, { "text": "größtmöglicher", "start": 429, "end": 443, "id": 59 }, { "text": "Sicherheit", "start": 444, "end": 454, "id": 60 }, { "text": "ausgeschlossen", "start": 455, "end": 469, "id": 61 }, { "text": "ist", "start": 470, "end": 473, "id": 62 }, { "text": ".", "start": 473, "end": 474, "id": 63 }, { "text": "Mit", "start": 475, "end": 478, "id": 64 }, { "text": "BGBl", "start": 479, "end": 483, "id": 65 }, { "text": "II", "start": 484, "end": 486, "id": 66 }, { "text": "2020", "start": 487, "end": 491, "id": 67 }, { "text": "/", "start": 491, "end": 492, "id": 68 }, { "text": "609", "start": 492, "end": 495, "id": 69 }, { "text": "wurde", "start": 496, "end": 501, "id": 70 }, { "text": "nun", "start": 502, "end": 505, "id": 71 }, { "text": "der", "start": 506, "end": 509, "id": 72 }, { "text": "Zeitraum", "start": 510, "end": 518, "id": 73 }, { "text": ",", "start": 518, "end": 519, "id": 74 }, { "text": "in", "start": 520, "end": 522, "id": 75 }, { "text": "dem", "start": 523, "end": 526, "id": 76 }, { "text": "Freistellungen", "start": 527, "end": 541, "id": 77 }, { "text": "von", "start": 542, "end": 545, "id": 78 }, { "text": "Arbeitnehmern", "start": 546, "end": 559, "id": 79 }, { "text": "mit", "start": 560, "end": 563, "id": 80 }, { "text": "einem", "start": 564, "end": 569, "id": 81 }, { "text": "COVID-19-Risiko-Attest", "start": 570, "end": 592, "id": 82 }, { "text": "möglich", "start": 593, "end": 600, "id": 83 }, { "text": "sind", "start": 601, "end": 605, "id": 84 }, { "text": ",", "start": 605, "end": 606, "id": 85 }, { "text": "bis", "start": 607, "end": 610, "id": 86 }, { "text": "zum", "start": 611, "end": 614, "id": 87 }, { "text": "Ablauf", "start": 615, "end": 621, "id": 88 }, { "text": "des", "start": 622, "end": 625, "id": 89 }, { "text": "31.", "start": 626, "end": 629, "id": 90 }, { "text": "3.", "start": 630, "end": 632, "id": 91 }, { "text": "2021", "start": 633, "end": 637, "id": 92 }, { "text": "verlängert", "start": 638, "end": 648, "id": 93 }, { "text": ".", "start": 648, "end": 649, "id": 94 } ] }] ``` This particular doc work's but there's to many to find the one that fails. I'm not sure what to do at this point and I'm honestly fonsued as to why this would even happen. I'm providing the entities with character-idx so why would there be a mismatch? Any help would be much appreciated 🙂 ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [June 25, 2021, 3:08am UTC](https://support.prodi.gy/t/mismatched-tokenization-on-ner/4353/2 "2021-06-25T03:08:22Z")
Hi! Are you able to find that particular example it fails on in your data? The basic problem here is this: the character offsets in your annotations are referring to spans of texts that do not map to the token boundaries produced by the tokenizer. This means that an NER model, which predicts token-based tags, couldn't be updated with this information and learn anything from it, because it will nevr actually get to see those tokens at runtime. For instance, imagine you have a text like `"AB C"`, which the tokenizer will split into `["AB", "C"]`. If your data annotates a span describing the character offsets for `A`, this will be a problem, because there isn't a token for `A` that can be labelled. So it comes down to finding those cases in your data so you can see what the main problems are. I've posted a script on [this thread](https://support.prodi.gy/t/anotation-task-format-for-ner-manual-interface/1501/4) that you can use to find mismatched character offsets programmatically: > [@Anotation task format for ner\_manual interface](https://support.prodi.gy/t/anotation-task-format-for-ner-manual-interface/1501/4): > > An easy way to do this is to use spaCy’s [`Doc.char_span` method](https://spacy.io/api/doc#char_span), which creates a token span from character offsets. If the character offsets don’t align to the tokens, it returns `None`. So you can do something like this: > > ```python > nlp = spacy.load("en_core_web_sm") # or other model > > for example in examples: # your existing examples > doc = nlp(example["text"]) > for span in example["spans"]: > char_span = doc.char_span(span["start"], span["end"]) > if char_span is None: # start and end don't map to tokens > print("Misaligned tokens", example["text"], span) > > ``` Sometimes, the problem can be caused by basic preprocessing issues and leading/trailing whitespace, or even off-by-one errors in the character offsets (if you've generated them programmatically). Those can be usually be fixed pretty easily in a preprocessing step. In other cases, it can point to tokenization settings that you might want to adjust to better fit the data you're working with. For instance, a string like `"hello.world"`, which the tokenizer preserves as one token `["hello.world"]`, but which you might want to split into 3 tokens `["hello", ".", "world"]`. If there a common patterns like this in your data, you can adjust the tokenization rules to be stricter and split more by modifying the punctuation rules: [Linguistic Features · spaCy Usage Documentation](https://spacy.io/usage/linguistic-features#native-tokenizer-additions) In general, it's good to tackle this early on and make sure you have a tokenizer in place that produces tokens that match the spans you're looking to annotate and predict – otherwise, you can easily end up with data that your model can't learn from. ---
**Author:** ![lnatprodigy](https://avatars.discourse-cdn.com/v4/letter/l/6f9a4e/32.png) [@lnatprodigy](https://support.prodi.gy/u/lnatprodigy)\ **Post date:** [June 25, 2021, 8:38am UTC](https://support.prodi.gy/t/mismatched-tokenization-on-ner/4353/3 "2021-06-25T08:38:43Z")
Thank you so much @ines ! That explains the problem very nicely. Since I already had a basic model trained in spacy v3, I'm now using the nightly version of prodigy, with that model in the loop, instead of feeding it pre-annotated data, and it works perfectly 🙂 I still have some tokenization issues (like dots at the end of text not being a separate token and such) and your post will be very helpful in solving those.