# Spans not lining up with text tokens, autopopulate NER

**URL:** <https://support.prodi.gy/t/spans-not-lining-up-with-text-tokens-autopopulate-ner/3949>\
**Category:** Uncategorized\
**Tags:** usage, ner, solved\
**Created:** [February 26, 2021, 8:33pm UTC](https://support.prodi.gy/t/spans-not-lining-up-with-text-tokens-autopopulate-ner/3949 "2021-02-26T20:33:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![aanifh](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/aanifh/32/1831_2.png) [@aanifh](https://support.prodi.gy/u/aanifh)\
**Post date:** [February 26, 2021, 8:33pm UTC](https://support.prodi.gy/t/spans-not-lining-up-with-text-tokens-autopopulate-ner/3949/1 "2021-02-26T20:33:39Z")

</div>

Hi,  
I have created an auto-populated file, but with "text", "tokens" and spans" fields". But when I upload it to Prodigy, the spans are misaligned with the text, from the 2nd sentence onward. However, the "ID", "start", "end" etc numbers seem to align to me  
Do you have any advice on how to fix this? (I know the "ws" field is off, but even if I don't include it it doesn't seem to matter) Thank you 😃

`{"text": "I'm standing here with you, why won't you move?", "tokens": [{"text": "I'm", "start": 38, "end": 40, "id": 13, "ws": "true"}, {"text": "standing", "start": 41, "end": 48, "id": 14, "ws": "true"}, {"text": "here", "start": 49, "end": 52, "id": 15, "ws": "true"}, {"text": "with", "start": 53, "end": 56, "id": 16, "ws": "true"}, {"text": "you", "start": 57, "end": 59, "id": 17, "ws": "true"}, {"text": ",", "start": 60, "end": 60, "id": 18, "ws": "true"}, {"text": "why", "start": 61, "end": 63, "id": 19, "ws": "true"}, {"text": "won't", "start": 64, "end": 68, "id": 20, "ws": "true"}, {"text": "you", "start": 69, "end": 71, "id": 21, "ws": "true"}, {"text": "move", "start": 72, "end": 75, "id": 22, "ws": "true"}, {"text": "?", "start": 76, "end": 76, "id": 23, "ws": "true"}], "spans": [{"text": "you", "start": 57, "end": 59, "token_start": 17, "token_end": 17, "label": "PRON"}, {"text": "you", "start": 69, "end": 71, "token_start": 21, "token_end": 21, "label": "PRON"}]}`

 ![Screenshot 2021-02-26 at 20.26.47](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/4/4fc793469248ed9e0578d598b85ae427606a4a80.png) ![Screenshot 2021-02-26 at 20.26.52](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/8/8bd0b50da4949ebf90c99efd61685c158aa09878.png)

And this happened when I tried to annotate one of the spans:

 ![Screenshot 2021-02-26 at 21.08.21](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/b/b820b7a38815014a86ea4478663b5f4dd41ecb20.png)

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [February 27, 2021, 12:10am UTC](https://support.prodi.gy/t/spans-not-lining-up-with-text-tokens-autopopulate-ner/3949/2 "2021-02-27T00:10:54Z")

</div>

Hi! I was just looking at the example you posted and it seems like the start/end offsets and IDs are off? For example, this is the first token:

```json
{"text": "I'm", "start": 38, "end": 40, "id": 13, "ws": "true"}

```

But it defines character `38` as its start offset into the `"text"` (which should probably be `0`), and `13` as its ID. Did you maybe generate them from sentences? If so, you can adjust the standalone examples by subtracting the sentence start and token offsets – so if your sentence starts at character 38, the first token's offset would be `token.start - 38 = 0`, and so on.

---

<div class="post-metadata">

**Author:** ![aanifh](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/aanifh/32/1831_2.png) [@aanifh](https://support.prodi.gy/u/aanifh)\
**Post date:** [March 3, 2021, 1:17pm UTC](https://support.prodi.gy/t/spans-not-lining-up-with-text-tokens-autopopulate-ner/3949/3 "2021-03-03T13:17:02Z")

</div>

Ahhh I understand now, I was doing a total tally from across the document 🙈 It's per-each "line", I've adjusted my code and it's all working really well now, thank you very much! 😃