Hi Prodigy!
I trained a texcat model with 3 labels which are mutually exclusive. And then, following the advice for improving a trained model I ran prodigy.teach against the three labels. So after the annotation process it gave me(in a dataset) like: this label correct? yes or no. Since I have 3 labels my question is this: how should I train/retrain the model with the format above for the three labels?
When I just ran the prodigy.train textcat command on the taught dataset it gave me something weird::
Baseline accuracy: 37.997
1 553.92 18.882
2. 82.93 18.882
3 24.42 18.882
4 11.30 18.882
5 5.67 18.882
6 3.48 18.882
7 2.18 18.882
8 1.49 18.882
...
So you see the F-Score does not change(it's always the same) as the loss gets to 0.
Maybe I need to manually extract the accepted answers but how should I treat the rejected ones since I have three labels to train against.
Please, advise
Hi! This all sounds good – if you have one dataset with all annotations, you can use that to update your model. Prodigy will take care of merging your annotations automatically and consider the accept/reject decisions accordingly, so you won't have to do anything here.
If your classes are mutually exclusive, make sure to set --textcat-exclusive on the CLI.
Yes, it looks like your model isn't learning anything. There are different reasons why this might happen – it depends on the data, categories and number of examples, and of course the examples and number of examples used for evaluation.
Definitely try running it again with --textcat-exclusive. You can also run data-to-spacy to export a JSON-formatted training file you can use with spacy train and that you can more easily inspect, so you can see how the annotations were merged.
Hi Ines! Thanks for a quick reply, I will investigate more following your advice!
I exported the dataset with the data-to-spacy command to a file. And when I was training a model with the file it shew me the following in the beginning of the process:
"Some textcat training instances do not have exactly one positive
label. Modifying training options to include the flag '--textcat-multilabel' for
classes that are not mutually exclusive. "
I am wondering how the textcat.teach logic works:
I have 3 mutually exclusive labels(say, Positive, Negative, Neutral). I run the prodigy textcat.teach command(no --textcat-exclusive option is available for it). When Prodigy suggests a next sample to annotate it is like: "What an amazing task!" Neutral -> Yes, No, Skip, Back.
When I click "No" how does Prodigy treat it? Because if it is not Neutral, it is either Positive or Negative since the labels are mutually exclusive, right? Or will Prodigy ask me one more time until it has got a Yes answer on the sample?
When I exported the db dataset to a spacy file, its contents was like this: ...Neutral:0, Positive:0, Negative: 0. So the spacy message above makes sense.
Just a bit confused. Please, advise more 
The --textcat-exclusive flag is available when you train your model on all annotations using prodigy train.
When you reject a binary suggestion for label A, all we know is that label A doesn't apply, but nothing about the two other labels. So when the model is updated in the loop, the goal is to update it with the best possible information based on what we know. For example, if the probabilities for the 3 labels are [0.5, 0.2, 0.3] and we know that the first label doesn't apply and should be 0.0, we can redistribute the probabilities based on the existing predictions for the remaining categories. So in this case, we'd update with [0.0, 0.4, 0.6], i.e. with the information that label A doesn't apply, without asserting anything about labels B and C. My slides here show an example of this for NER, but the idea for text classification is very similar: https://speakerdeck.com/inesmontani/belgium-nlp-meetup-rapid-nlp-annotation-through-binary-decisions-pattern-bootstrapping-and-active-learning?slide=12
Thanks Ines, it all looks good to me!