# Confused about the structure of spans in NER examples

**URL:** <https://support.prodi.gy/t/confused-about-the-structure-of-spans-in-ner-examples/7048>\
**Category:** Uncategorized\
**Created:** [January 23, 2024, 2:54pm UTC](https://support.prodi.gy/t/confused-about-the-structure-of-spans-in-ner-examples/7048 "2024-01-23T14:54:49Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![akimotode](https://avatars.discourse-cdn.com/v4/letter/a/ccd318/32.png) [@akimotode](https://support.prodi.gy/u/akimotode)\
**Post date:** [January 23, 2024, 2:54pm UTC](https://support.prodi.gy/t/confused-about-the-structure-of-spans-in-ner-examples/7048/1 "2024-01-23T14:54:49Z")

</div>

Esteemed prodigy experts,

I have played with the get\_dataset\_examples functionality and I am receiving two different formats of examples. Could someone help to clear up why this exists?

My code is very simple: I am handing over the name of a dataset and collect the examples in a list.

```python
       db = connect()

        # Set up the result list
        lst_examples = []

        examples = db.get_dataset_examples(dataset_name)
        for example in examples:
            lst_examples.append(example)

        db.close()

```

This all works as expected, but when I expect the spans in the examples, I get two different formats of spans. Two have the actual text of the span, a source and an input hash - the third one does not.

 ![Capture](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/2/2c4da144d344a451b50b2f601e7b0f00db15af97.png)

I am using ner.correct in this case. My hunch is that the "longer" format is created by the prediction model while the "shorter" format is created by a manual action. Could that be true and if so, why is that?

I've tried peaking into the db itself but I don't think that will clear this up.

Many thanks for any help!  
Kai

---

<div class="post-metadata">

**Author:** ![akimotode](https://avatars.discourse-cdn.com/v4/letter/a/ccd318/32.png) [@akimotode](https://support.prodi.gy/u/akimotode)\
**Post date:** [January 23, 2024, 3:03pm UTC](https://support.prodi.gy/t/confused-about-the-structure-of-spans-in-ner-examples/7048/2 "2024-01-23T15:03:57Z")

</div>

Oh, in case this is helpful/or needed: I am on the latest prodigy, Python 3.11 and Windows

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [January 24, 2024, 11:20am UTC](https://support.prodi.gy/t/confused-about-the-structure-of-spans-in-ner-examples/7048/3 "2024-01-24T11:20:01Z")

</div>

Hi @akimotode and welcome to the forum 👋

You're definitely on the right track! The annotations with extra keys come from the model. The `source` field gives you the name of the NLP pipeline that added the annotations. Annotations you add manually via Prodigy UI won't have these extra keys.

The manual annotations contain just the minimum viable information that are required to train the model. The extra information in the case of model annotated spans comes "for free" from the spaCy pipeline, concretely from the [Span](https://spacy.io/api/span) attributes of recognized entites so the recipe just adds it.

---

<div class="post-metadata">

**Author:** ![akimotode](https://avatars.discourse-cdn.com/v4/letter/a/ccd318/32.png) [@akimotode](https://support.prodi.gy/u/akimotode)\
**Post date:** [January 24, 2024, 12:48pm UTC](https://support.prodi.gy/t/confused-about-the-structure-of-spans-in-ner-examples/7048/4 "2024-01-24T12:48:54Z")

</div>

Thank you Magda for the very quick answer! At least now I know I haven't managed to shoot myself in the foot somehow...

If I may, a quick feature request: I find it really useful that the model adds the "text" information. I can get this via tokens anyway, but it is a bit tricky (and it is not easily visible during debugging). If there was a simple way to add "text" to the manual annotation, it would be helpful for my use case. It's definitely only a convenience change, I can already do what I want to do.

My use case: I want to iterate over all examples and output a report which spans have been tagged with which label across a group of datasets. Sort of "automated meta" quality control across datasets and raters...

Thanks,  
Kai

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [January 26, 2024, 1:04pm UTC](https://support.prodi.gy/t/confused-about-the-structure-of-spans-in-ner-examples/7048/5 "2024-01-26T13:04:14Z")

</div>

Hi @akimotode,

Thanks for the suggestion! The thinking here is that Prodigy should not really modify the annotations coming from the user as they can contain custom fields and values.

The easiest way to add `text` to the manual annotation would be either to run a simple postprocessing script on the annotated dataset (which is what you're probably doing) or to create a custom NER recipe and implement span processing function via [`before_db`](https://prodi.gy/docs/custom-recipes#before_db) callback.  
Here's a simplified version of such recipe:

```python
from pathlib import Path
from typing import Any, Dict

import prodigy
import spacy
from prodigy.components.stream import get_stream
from prodigy.components.preprocess import add_tokens

def before_db(examples):
    for example in examples:
        for span in example['spans']:
            start = span['start']
            end = span['end']
            span['text'] = example['text'][start:end]
    return examples

@prodigy.recipe(
    "ner.with_text",
    dataset=("Dataset to save annotations to", "positional", None, str),
    lang=("language for the tokeniser", "positional", None, str),
    source=("Data to annotate", "positional", None, str),
    labels=("comma separated sequence of labels", "option", "l", str),
)
def ner_with_text(dataset: str, lang: str, source: Path, labels: str) -> Dict[str, Any]:
    nlp = spacy.blank(lang)
    labels = labels.split(",")
    stream = get_stream(source, rehash=True, input_key="text", dedup=True)
    stream = stream.apply(add_tokens,nlp=nlp, stream=stream)

    return {
        "dataset": dataset,
        "stream": stream,
        "view_id": "ner_manual",
        "before_db": before_db,
        "config": {
            "lang": lang,
            "labels": labels,
            "batch_size": 1
        },
    }

```

Now if you test it with:

```python
{"text": "hi my name is Kai"}

```

You'll see that the DB record for this example will contain the text field:

```python
"spans": [
    {
      "start": 14,
      "end": 17,
      "token_start": 4,
      "token_end": 4,
      "label": "name",
      "text": "Kai"
    }

```

---

<div class="post-metadata">

**Author:** ![akimotode](https://avatars.discourse-cdn.com/v4/letter/a/ccd318/32.png) [@akimotode](https://support.prodi.gy/u/akimotode)\
**Post date:** [January 30, 2024, 2:07pm UTC](https://support.prodi.gy/t/confused-about-the-structure-of-spans-in-ner-examples/7048/6 "2024-01-30T14:07:37Z")

</div>

Thank you Magda,

This is an excellent answer!