# Annotating an extractive QA dataset à la SQuAD

**URL:** <https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125>\
**Category:** Uncategorized\
**Tags:** usage, custom, front-end, solved\
**Created:** [October 21, 2019, 2:18pm UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125 "2019-10-21T14:18:10Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![vitojph](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vitojph/32/350_2.png) [@vitojph](https://support.prodi.gy/u/vitojph)\
**Post date:** [October 21, 2019, 2:18pm UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/1 "2019-10-21T14:18:10Z")

</div>

Hi all,

I'm trying to use prodigy to create a dataset to train an extractive Question Answering system. As in SQuAD, each sample contains a `question`, a `context` (ie. a paragraph in natural language, related to the question and likely to contain a possible answer) and the specific `answer`, extracted directly from the context. Notice that the `answer` is a substring of the `context`.

I have already collected a set of pairs of questions and contexts. I need the human annotators to select the span of the context that corresponds with the answer, if any.

So, my initial idea was to combine the functionality of `ner.manual` to tokenize the context and select the span of the answer along with some kind of custom HTML view to show for each sample both the question and the context. Is there any better approach? Any hints to tackle this? Thanks in advance,

EDIT: I've found this [custom question answering recipe](https://github.com/explosion/prodigy-recipes/blob/master/other/question_answering.py) but it doesn't fit my needs.

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 22, 2019, 10:10am UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/2 "2019-10-22T10:10:13Z")

</div>

> [@vitojph](#):
>
> So, my initial idea was to combine the functionality of `ner.manual` to tokenize the context and select the span of the answer along with some kind of custom HTML view to show for each sample both the question and the context. Is there any better approach? Any hints to tackle this? Thanks in advance,

Hi! This is probably quite similar to what I would have suggested 🙂 So if I understand the data correctly, the only additional piece of information you need to display is the question, right? And the context and answer are already covered by the text plus pre-highlighted span you'd use in the `ner_manual` interface?

Assuming your incoming tasks look like this:

```json
{"question": "...", "text": "...", "tokens": [...], "spans": [...]}

```

You could then use some custom JavaScript to add the `"question"` (available as `window.prodigy.content.question` in this case) before the main content (the highlightable text with the pre-highlighted answer).

```javascript
document.addEventListener('prodigyupdate', event => {
    const container = document.querySelector('.prodigy-content');
    container.prepend(window.prodigy.content.question)
})

```

On a related note, this upcoming feature will make these combined interfaces a lot easier. You could then just have two blocks: HTML and the manual NER interface.

[https://twitter.com/\_inesmontani/status/1186275862527635456](https://twitter.com/_inesmontani/status/1186275862527635456)

---

<div class="post-metadata">

**Author:** ![vitojph](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vitojph/32/350_2.png) [@vitojph](https://support.prodi.gy/u/vitojph)\
**Post date:** [October 23, 2019, 7:50am UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/3 "2019-10-23T07:50:26Z")

</div>

Thanks @ines. As you suggest, I'll try to modify the Javascript to display the question 🙂

BTW, I saw your tweet and the new feature looks cool. Looking forward to use it.

---

<div class="post-metadata">

**Author:** ![vitojph](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vitojph/32/350_2.png) [@vitojph](https://support.prodi.gy/u/vitojph)\
**Post date:** [October 24, 2019, 9:25am UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/4 "2019-10-24T09:25:14Z")

</div>

Hi, again!

I found a solution to show both the `question` and the `context`, and to able to select the answer span.

![Captura%20de%20pantalla%20de%202019-10-24%2011-11-09](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/a/a3f0109bad7359eadb5cce7dd0814188b3698948.png)

![Captura%20de%20pantalla%20de%202019-10-24%2011-10-28](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/4/4112495e0e198a39a3735c9c9c4f1f16e0a561dc.png)

Just add this custom Javascript to your `~/.prodigy/prodigy.json` file:

```
{
"javascript": "document.addEventListener('prodigyupdate', event => {const container = document.querySelector('.prodigy-title'); container.innerHTML=window.prodigy.content.question; });document.addEventListener('prodigymount', event => {const container = document.querySelector('.prodigy-title'); container.innerHTML=window.prodigy.content.question; })"
}

```

And my custom recipe looks like:

```
import prodigy
import spacy
from prodigy.components.loaders import JSONL
from prodigy.components.preprocess import add_tokens
from prodigy.util import split_string

@prodigy.recipe(
    "qa",
    dataset=("The dataset to use", "positional", None, str),
    spacy_model=("The base model", "positional", None, str),
    source=("The source data as a JSONL file", "positional", None, str),
    label=("One or more comma-separated labels", "option", "l", split_string),
)
def qa(dataset, spacy_model, source, label="answer_span"):
    """Custom recipe to annotate a dataset to train and evalute an extractive Question Answering system"""

    # load the source dataset, made of samples containing question and text pairs
    stream = JSONL(source)
    # load the spaCy model
    nlp = spacy.load(spacy_model)
    # and tokenize the text
    stream = add_tokens(nlp, stream)

    return {
        "view_id": "ner_manual",
        "dataset": dataset,
        "stream": stream,
        "config": {"lang": nlp.lang, "label": label, "labels": label},
    }

```

Thanks a lot!

---

<div class="post-metadata">

**Author:** ![vitojph](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vitojph/32/350_2.png) [@vitojph](https://support.prodi.gy/u/vitojph)\
**Post date:** [October 24, 2019, 9:43am UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/5 "2019-10-24T09:43:18Z")

</div>

@ines Please let me know if I can send you a PR to [https://github.com/explosion/prodigy-recipes](https://github.com/explosion/prodigy-recipes) to add this recipe

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 25, 2019, 9:02am UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/6 "2019-10-25T09:02:06Z")

</div>

@vitojph Thanks and yes, that'd be cool! You can add the `"javascript"` to the `"config"` returned by the recipe – this way, everything can fit into one file.

---

<div class="post-metadata">

**Author:** ![vitojph](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vitojph/32/350_2.png) [@vitojph](https://support.prodi.gy/u/vitojph)\
**Post date:** [October 25, 2019, 6:29pm UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/7 "2019-10-25T18:29:46Z")

</div>

@ines I don't seem to have permissions to push new branches to the repo. Can you please give me access? My GitHub account is `vitojph`.

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 26, 2019, 11:41am UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/8 "2019-10-26T11:41:03Z")

</div>

@vitojph Just fork the repo and [submit a PR from your fork](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork) 🙂

---

<div class="post-metadata">

**Author:** ![sanchez-castro](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sanchez-castro/32/2788_2.png) [@sanchez-castro](https://support.prodi.gy/u/sanchez-castro)\
**Post date:** [February 24, 2022, 6:54pm UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/9 "2022-02-24T18:54:27Z")

</div>

Hi @vitojph and @ines

My name is Adrian, working on a Squad Q&A project at Stanford.

I have seen that this recipe is available in the repo of prodigy recipes.  
Would you confirm the json schema that this recipe supports?

From the function documentation it states:

> Annotate question/answer pairs with a custom HTML interface. Expects an  
> input file with records that look like this:  
> {"question": "What color is the sky?", "question\_answer": "blue"}

Where is the context?  
Should I use the key **"text"**?  
Shouldn't it be like:  
{"question": "What color is the sky?", **"text": \<paragraph\>** , "question\_answer": "blue"}

Thanks in advance.

p.s. why are the "others" recipes not part of the basic installation?

Adrian

---

<div class="post-metadata">

**Author:** ![sanchez-castro](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sanchez-castro/32/2788_2.png) [@sanchez-castro](https://support.prodi.gy/u/sanchez-castro)\
**Post date:** [February 24, 2022, 8:23pm UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/10 "2022-02-24T20:23:13Z")

</div>

It would be great to have the recipe instructions in the official documentation [website](https://prodi.gy/docs/recipes) too 🙂

---

<div class="post-metadata">

**Author:** ![sanchez-castro](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sanchez-castro/32/2788_2.png) [@sanchez-castro](https://support.prodi.gy/u/sanchez-castro)\
**Post date:** [February 25, 2022, 3:22am UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/11 "2022-02-25T03:22:52Z")

</div>

🤔

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [February 25, 2022, 11:38am UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/12 "2022-02-25T11:38:45Z")

</div>

> [@sanchez-castro](#):
>
> Where is the context?  
> Should I use the key **"text"**?

The example recipe here is intended to do a binary review of question/answer pairs without context, but you could easily include the context as well by modifying the code and including the context in the HTML template: [https://github.com/explosion/prodigy-recipes/blob/master/other/question\_answering.py](https://github.com/explosion/prodigy-recipes/blob/master/other/question_answering.py)

Alternatively, the example recipe shared above takes a slightly different approach

> [@sanchez-castro](#):
>
> p.s. why are the "others" recipes not part of the basic installation?
> 
> It would be great to have the recipe instructions in the official documentation [website](https://prodi.gy/docs/recipes) too 🙂

The `prodigy-recipes` repo is a collection of templates for developing custom recipes with Prodigy. It includes simplified versions of some of the built-in recipes that can be adapted, as well as examples of different use cases and contributions from the community. None of these recipes are included verbatim in the default installation – they're templates for developing your own, or specific examples that are not general-purpose enough to be useful as a built-in recipe, but still interesting.

P.S.: It's not really helpful to bump a thread multiple times within a short time frame, especially not with just single emojis. This makes it a lot harder for us (and othe readers) to follow new posts.

---

<div class="post-metadata">

**Author:** ![sanchez-castro](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sanchez-castro/32/2788_2.png) [@sanchez-castro](https://support.prodi.gy/u/sanchez-castro)\
**Post date:** [March 6, 2022, 1:34am UTC](https://support.prodi.gy/t/annotating-an-extractive-qa-dataset-a-la-squad/2125/13 "2022-03-06T01:34:07Z")

</div>

> [@ines](#):
>
> The `prodigy-recipes` repo is a collection of templates for developing custom recipes with Prodigy. It includes simplified versions of some of the built-in recipes that can be adapted, as well as examples of different use cases and contributions from the community. None of these recipes are included verbatim in the default installation – they're templates for developing your own, or specific examples that are not general-purpose enough to be useful as a built-in recipe, but still interesting.

thanks! but one more emoji  
🤪