In the Recipes page, the example under textcat.teach is multilabel. I tested it with the news_headlines.jsonl sample and with the addition of a few patterns (one for each of the labels Technology,Politics,Economy,Entertainment based on the text in the sample). It seems to work (as in "it doesn't-break0-and-it-even-trains-afterwards), but...
a) Does it actually work out-of-the-box (from some old questions here I got the impression that it was not supposed to) or is the underlying code supposed to work with a single label only (and it just happens to not break with multi labels)?
b) If yes (works), I'm wondering what's the advantage over the manual recipe when dealing with many possible non-exclusive labels? From my understanding, by going over the texts one label at a time (as recommended by the Docs if more than 10-20 labels), I'd need to maybe do N_labels passes (e.g., 4 in news_headlines example) for each text. If not, for any text that got an accept/reject less than N, all other "missing" labels would have to be assumed as, say, False. But with "teach" at least that wouldn't be random as it focus first on the cases the model is less sure about it, is that it?
Context: I'm planning on labelling a few dozen thousands of sentences with about 30 non-exclusive labels, and thus trying to plan on the best approach (as having 30 click options each time doesn't sound fun).
just for extra clarity. the textcat.teach recipe can be tweaked by customising the prodigy.json file. In particular, if you have exclusive labels you may want to set the choice_auto_accept flag.
Now to answer your questions.
a) Does it actually work out-of-the-box
It's a bit hard for me to understand what you are referring to? Is there a command that you ran that behave the way you expected? If so, could you expand?
As general annotation advice: do you really need to have all 30 labels immediately? Having an interface with 30 labels will be a cognitive burden to the person who is annotating and may lead to lower data quality. If it's possible, I'd recommend starting with fewer labels. This has the benefit of higher data quality, but it might be faster to start with a model that predicts just 4 labels initially. You're always able to add more labels later.
If you can shed some more details on what task you'll be dealing with I might be able to give more directed advice.
Thank you for the reply. My labels are non-exclusive but thanks for the tip.
Firstly, I'm basically trying to understand how textcat.teach (without any customization) works in a multilabel context. If we start with the textcat.manual recipe, things are pretty straightforward: if you have 4 categories (A, B, C, D), prodigy shows all 4 to you, and for one text if you click on, say, two of them (A, B), you have {A:True, B:True, C:False, D:False}. When I train a model, I'll be telling it that "hey, there is A and B in text1, but not C and D". Now, with textcat.teach, prodigy will only show one category at a time, say A. Thus my output is basically {A:True/False}. But what about B, C, and D? Will prodigy train for example assume them as False if the text is not shown again with another label?
Secondly, on my use case, I did start with a smaller number of categories, but it was not granular enough. ~30 was the sweet spot after several other attempts (disregarding annotation difficulties). It's a similar use case to the goEmotions dataset from one of your videos. So, if you were to label for the goEmotions dataset, would the textcat.teach recipe be useful?
The spaCy models for text classification (which is what `prodigy train uses under the hood) are able to deal with each output independently. That means that once you annotate a sentence for label A that output will have more data to learn from. That's it! There's no information assumed for outputs B, C and D in this case. Only the labels that have been provided will be used to train the outputs of the spaCy model.
I'm assuming you've seen this video, in which you can see that I'm really just checking one label. It's much easier for me to annotate one emotion at a time, which is what I'm doing there. I can also add more emotions to the dataset later if I wanted to though.