# Providing NER token spans only (no character offsets)

**URL:** <https://support.prodi.gy/t/providing-ner-token-spans-only-no-character-offsets/1886>\
**Category:** Uncategorized\
**Tags:** usage, spacy, best-practices\
**Created:** [August 9, 2019, 6:00pm UTC](https://support.prodi.gy/t/providing-ner-token-spans-only-no-character-offsets/1886 "2019-08-09T18:00:58Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![henrye](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/henrye/32/569_2.png) [@henrye](https://support.prodi.gy/u/henrye)\
**Post date:** [August 9, 2019, 6:00pm UTC](https://support.prodi.gy/t/providing-ner-token-spans-only-no-character-offsets/1886/1 "2019-08-09T18:00:58Z")

</div>

Hi!

I'm lazy (!) and providing character spans in the JSONL for NER annotation seems a little tedious. I've looked at the recipe but couldn't quite figure out, is it possible to just provide tokens spans only?

For example

```python
{
    "text": "Hello Apple",
    "tokens": [
        { "text": "Hello", "id": 0 },
        { "text": "Apple", "id": 1 }
    ],
    "spans": [{ "label": "ORG", "token_start": 1, "token_end": 1 }]
}

```

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [August 10, 2019, 11:29am UTC](https://support.prodi.gy/t/providing-ner-token-spans-only-no-character-offsets/1886/2 "2019-08-10T11:29:00Z")

</div>

Hi! This isn't supported out-of-the-box, because then Prodigy would have to be in charge of aligning the tokenization. This can get pretty tricky, so at the moment, this shouldn't be something that the library does silently under the hood.

If you want, you can write a script like this using spaCy that takes the `tokens` to construct a new `Doc` object and then gets the character offsets of the spans into the `doc.text` by checking the span at `doc[start:end]`. Here's an example:

```python
from spacy.tokens import Doc
from spacy.lang.en import English
nlp = English() # mostly need this for the vocab

examples = []
for eg in your_data_here:
    words = [token["text"] for token in eg["tokens"]]
    doc = Doc(nlp.vocab, words=words)
    spans = []
    for span in eg["spans"]:
        doc_span = doc[span["token_start"]:span["token_end"] + 1]
        spans.append({
            "start": doc_span.start_char,
            "end": doc_span.end_char,
            "label": span["label"]
        })
     # We don't need tokens here anymore because we know
     # it matches spaCy's tokenization
     examples.append({"text": doc.text, "spans": spans})

```

However, there's one thing that this approach cannot solve for you: If you don't have character offsets and your tokenization doesn't preserve whitespace (like in your example), we have no way of knowing whether tokens are followed by a whitespace character or not and have to assume that they are. So the tokens `"Hello", "Apple", "!"` would become `"Hello Apple !"`. If you have the whitespace information, you can pass in a list of boolean values as the `spaces` argument when constructing a `Doc`, e.g. `spaces=[True, False, False]`.

If you don't have the whitespace information, you can try to reconstruct it from the original text by matching it up with the token texts and checking for trailing whitespace characters. I had to do this for our [`spacy-stanfordnlp` wrapper](https://github.com/explosion/spacy-stanfordnlp) and it looks something like this:

> <https://github.com/explosion/spacy-stanfordnlp/blob/aa7371165778cc281536491f572eb3ff71f3c5f3/spacy_stanfordnlp/language.py#L148-L170>

Finally, this approach is still going to fail if your tokenizer is destructive and doesn't follow a policy of preserving the original text. For instance, some tokenizers may output `"I", "am"` for the string `"I'm"`. There are algorithms for handling this, but if your main objective is that you're lazy, I'm not sure you want to go down that rabbit hole 😉

**TL;DR:** If your tokenization preserved whitespace information (whether the tokens are followed by a space or not) and is otherwise non-destructive, or if you don't care that much about preserving the original `"text"`, use spaCy to auto-add the character offsets for you. Otherwise... it's more difficult.

---

<div class="post-metadata">

**Author:** ![henrye](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/henrye/32/569_2.png) [@henrye](https://support.prodi.gy/u/henrye)\
**Post date:** [August 12, 2019, 4:56pm UTC](https://support.prodi.gy/t/providing-ner-token-spans-only-no-character-offsets/1886/3 "2019-08-12T16:56:36Z")

</div>

Ok awesome, thanks for the incredibly detailed guidance! I will give this a shot and see how it goes.

In this problem it's definitely simplified because all the inputs will be tokenized already and any new inputs will also be tokenized. Using the spacy code to get the character values seems easier than writing a new character counter, which is what I was hoping to avoid!