# From Choice annotations to binary annotations with Teach

**URL:** <https://support.prodi.gy/t/from-choice-annotations-to-binary-annotations-with-teach/1092>\
**Category:** Uncategorized\
**Tags:** usage, textcat, spacy\
**Created:** [December 29, 2018, 4:59pm UTC](https://support.prodi.gy/t/from-choice-annotations-to-binary-annotations-with-teach/1092 "2018-12-29T16:59:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nuno](https://avatars.discourse-cdn.com/v4/letter/n/65b543/32.png) [@nuno](https://support.prodi.gy/u/nuno)\
**Post date:** [December 29, 2018, 4:59pm UTC](https://support.prodi.gy/t/from-choice-annotations-to-binary-annotations-with-teach/1092/1 "2018-12-29T16:59:25Z")

</div>

I am trying to use Prodigy to annotate text documents with multiple classification labels (each observation can have more than 1 label). I used a custom recipe with the Choice interface to generate the first annotations with all labels.

Now, I would like to use the Teach recipe with active learning to annotate each of the labels individually. Lets say I have 4 labels: A, B, C, D. I will use Teach with each label at a time and I want Prodigy to suggest the documents which are more likely part of the respective label for me to accept or reject the annotation. I want to do this because it will be a faster way to annotate and a better use of Prodigy.

How can I go from my initial dataset of annotations created with the Choice interface to using Teach with a single label (e.g. label A) to generate more annotations? (and then repeating the process for label B, label C, label D)

Thanks!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [December 30, 2018, 11:08am UTC](https://support.prodi.gy/t/from-choice-annotations-to-binary-annotations-with-teach/1092/2 "2018-12-30T11:08:29Z")

</div>

Hi! This sounds like a good strategy 👍

> [@nuno](#):
>
> How can I go from my initial dataset of annotations created with the Choice interface to using Teach with a single label (e.g. label A) to generate more annotations? (and then repeating the process for label B, label C, label D)

Ideally, you probably want to use your existing annotations to pre-train a model and then use that as the base model for `textcat.teach` and improve it further. For this step, you might actually find it easier to train the text classifier [directly with spaCy](https://spacy.io/usage/training#section-textcat).

Once you've collected annotations using your custom recipe, you can export the data to a file:

```bash
prodigy db-out your_dataset > data.jsonl

```

Each entry will probably look something like this:

```json
{
    "text": "Some text",
    "options": [
        {"id": "LABEL_ONE", "text": "Label one"},
        {"id": "LABEL_TWO", "text": "Label two"}
    ],
    "accept": ["LABEL_TWO"]
}

```

To train the text classifier (and pretty much all other components) in spaCy, you need a text and a dictionary of the annotations – for the textcat component, that's the `"cats"` and the label values mapped to booleans (whether the label applies):

```python
training_data = [
    ("Some text", {"cats": {"LABEL_ONE": False, "LABEL_TWO": True}})
    # etc.
]

```

Getting this data from your chocie annotations should hopefully be a pretty straightforward data transformation. You can find more details and examples of spaCy's training loop [on this page](https://spacy.io/usage/training#example-textcat). You can start off with a blank model or with a pre-trained model (if you also want to keep the other pre-trained components like the tagger and parser).

Here's a quick summary of the training process that shows the concept. The [detailed example in the docs](https://spacy.io/usage/training#section-textcat) also shows how you can evaluate your model during training. Since you're only _pre-training_ it so you can improve it later, it's okay if the model isn't perfect

```python
nlp = spacy.load('en_core_web_sm') # or: nlp.blank('en') for blank model
textcat = nlp.create_pipe('textcat') # create new text classifier
for label in ('LABEL_ONE', 'LABEL_TWO'): # add labels
    textcat.add_label(label)
nlp.add_pipe(textcat) # add text classifier to the pipeline

with nlp.disable_pipes('tagger', 'parser', 'ner'): # only train text classifier
    nlp.begin_training() # initialize the weights
    for n in range(20): # number of iterations
        losses = {}
        random.shuffle(training_data) # shuffle converted data
        batches = spacy.util.minibatch(training_data) # batch up data
        for batch in batches:
            texts, annotations = zip(*batch)
            # update the model
            nlp.update(texts, annotations, drop=0.2, losses=losses)
       print("Losses", losses)

nlp.to_disk('/path/to/model') # save the model to a directory

```

The pre-trained model will be saved to `/path/to/model` and you can pass this directory in as the `spacy_model` argument when you run Prodigy. For example:

```bash
prodigy textcat.teach new_dataset /path/to/model your_data.jsonl --label LABEL_ONE

```

After you've collected some annotations, you can run Prodigy's `textcat.batch-train` with the custom pre-trained model as the base model and see how your binary annotations improve the model.

---

<div class="post-metadata">

**Author:** ![nuno](https://avatars.discourse-cdn.com/v4/letter/n/65b543/32.png) [@nuno](https://support.prodi.gy/u/nuno)\
**Post date:** [December 31, 2018, 4:23pm UTC](https://support.prodi.gy/t/from-choice-annotations-to-binary-annotations-with-teach/1092/3 "2018-12-31T16:23:19Z")

</div>

Hi Ines! Thanks a lot for your help. I have managed to follow your suggestion step-by-step until the end.

I converted the data, trained a multilabel model with spacy, saved the model to disk, and used it as a pre-trained model to help me annotate with `textcat.teach` for one of the labels.

I have 2 follow up questions:

1. If I generate binary annotations with `textcat.teach` and then do `textcat.batch-train` to train the multilabel model with those binary annotations, what exactly is the model being trained on? Does prodigy assume that all other labels are False?

2. While doing binary annotation for LABEL\_ONE, all the documents which `textcat.teach` suggests are negative examples (none of them are LABEL\_ONE). I would like to know how to make `textcat.teach` suggest the observations with highest probability of belonging to LABEL\_ONE so that I can generate more positive annotations? (instead of the most uncertain scores. I know there must be a lot of value in choosing the uncertain ones, but it seems that it’s not the best when you have an unbalanced multilabel dataset and you’re just getting started)

Thanks!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [January 2, 2019, 4:43pm UTC](https://support.prodi.gy/t/from-choice-annotations-to-binary-annotations-with-teach/1092/4 "2019-01-02T16:43:06Z")

</div>

> [@nuno](#):
>
> If I generate binary annotations with `textcat.teach` and then do `textcat.batch-train` to train the multilabel model with those binary annotations, what exactly is the model being trained on? Does prodigy assume that all other labels are False?

The default text classification model (via spaCy) assumes that categories are not mutually exclusive – so if you update the model with a text plus a category, the update is only performed for that label and all other labels are treated as unknown / missing values. Prodigy uses the same approach for binary NER annotations btw – [my slides here](https://speakerdeck.com/inesmontani/belgium-nlp-meetup-rapid-nlp-annotation-through-binary-decisions-pattern-bootstrapping-and-active-learning?slide=12) show an example of this process.

> [@nuno](#):
>
> I would like to know how to make `textcat.teach` suggest the observations with highest probability of belonging to LABEL\_ONE so that I can generate more positive annotations? (instead of the most uncertain scores. I know there must be a lot of value in choosing the uncertain ones, but it seems that it’s not the best when you have an unbalanced multilabel dataset and you’re just getting started)

Yeah, this sounds reasonable. The uncertainty sampling is performed by the `prefer_uncertain` sorter, which takes a stream of `(score, example)` tuples and yields `example`s. Under the hood, it uses an exponential moving average to determine whether to send out an example or not. Instead of `prefer_uncertain`, you can also use the `prefer_high_scores` sorter, which has the same API, but prioritises high scores.

So in `recipes/textcat.py`, you could update the `teach` recipe like this:

```python
from prodigy.components.sorters import prefer_high_scores

# in the recipe:
stream = prefer_high_scores(model(stream))

```

Our [`prodigy-recipes` repo](https://github.com/explosion/prodigy-recipes) also has a simplified version of the `textcat.teach` recipe with a bunch of comments explaining what's going on. So you might find this useful as well as a starting point to write your own custom version:

> <https://github.com/explosion/prodigy-recipes/blob/master/textcat/textcat_teach.py>

---

<div class="post-metadata">

**Author:** ![nuno](https://avatars.discourse-cdn.com/v4/letter/n/65b543/32.png) [@nuno](https://support.prodi.gy/u/nuno)\
**Post date:** [January 2, 2019, 5:12pm UTC](https://support.prodi.gy/t/from-choice-annotations-to-binary-annotations-with-teach/1092/5 "2019-01-02T17:12:29Z")

</div>

Thanks for the clarification! I’ll have to read more about that way of handling unknown labels, it seems quite interesting!

Regarding the second question, `prefer_high_scores` works. I’d suggest adding this as an argument for text.cat.teach.

Thanks! 🙂