Hi, I have a question about the textcat.teach recipe when using a model in the loop.
I first added some categories to a spanish base model.
Then I trained those categories with some labeled data.
I meant to use this partially trained model to bootstrap my labeling process in prodigy, but I’m seeing some unexpected behavior: The score seems to be from the first label if I run the example manually through spacy, not the one set by me. Surely this is not the intended output?
I’m launching prodigy from bash, like so
prodigy textcat.teach test models/output_model input.txt --label busqueda
Thanks! This was a bug in the meta attribute that displays the score. It doesn’t affect the way the examples are sorted, but it was changing the displayed score, giving the confusing results. We should have a new point release uploaded soon, which will include the fix.
The output I’m seeing from the above is a tuple with the scoring and the annotated example, as described in the documentation. The problem is that all the examples come with the same label, even if I pass multiple labels to the model, as in the code above.
In your example code, does the model you’re loading have a pre-trained text classifier component? If not, it’s possible that the model only asks about one label first and then adjusts accordingly. The built-in annotation recipes are also desgined to focus on one label at a time.
I think what Matt was referring to was a much more superficial mistake: The task’s "meta" property (which is what’s displayed in the UI) wasn’t always overwritten correctly, so the score that was displayed was the score from a different label. This didn’t have an impact on the actual scoring, only on what’s shown in the UI.
Yes, it does. It has 22 trained classes, some with thousands of examples. Sorry if the generic class names confused the situation, those are definitely not the names of the classes I’m using nor the actual code I tested. In the code the class names match.
Ok, what I posted in my follow-up is entirely different then. I understand that the process is designed to focus on one label at a time, and I intend to use it that way, but I tested the code above just to trace and narrow the problem down.
Notice the case I made before, where the scoring from the first output matches the scoring of the very first label of the second output. It applies to the meta value as @honnibal said, but it does for the other keys as well.