Change examples/annonations in prodigy DB from multiple to single

[ryanwesslen]

@joebuckle

Found the issue with the --verbose option. It was the 'text' field/property. Since we are using a custom recipe, it wasn't set correctly. It was set to the name of the audio file. That's why we weren't getting any good results for weeks now. Thank you very much for suggesting the --verbose option on the spacy debug data.

Awesome! I had forgotten about that option too. I saw it the docs and thought it may help (so I decided to mention in case ).

@joebuckle

I hope you don't mind me asking, but does using --base-model help in --textcat or --textcat-multilabel training?

Yes! They can. If you want to test, then try the en_core_web_lg first. You'll need the vectors which are in the md and lg models. You may not see a lot of improvement but it'll likely not add much for compute speed and memory.

There's sometimes a tendency to immediately go to transformers (en_core_web_trf), but they come with challenges (speed, memory, handling GPU). The speed/simplicity early on with the spaCy models can help you figure out problems with your annotation schemes, which can sometimes improve your model better than architecture (like vectors) or hyperparameters. In a 2018 talk, Matt called it the foundation of the "ML Hierarchy of Needs". Essentially, "categories that will be easy to annotate consistently, and easy for the model to learn."

Once you get promising results with your annotation schemes and performance, then you can test en_core_web_trf. You could also experiment with different textcat architectures.

Here's a related discussion (it was on ner but same idea of speed/accuracy trade-off for base-models applies):

Should I be using --base-model when training my model?

Hi @tomw! Likely a base model will help by providing word vectors but you'll want to turning off the ner component as your ner model will be trained from scratch. If you don't turn off the ner then you will be adding your new entity to the existing ner entities. This post shows how to do it (fyi some syntax has changed but general idea is the same): nlp = spacy.load("en_core_web_sm") nlp.remove_pipe("ner") # can remove other not used components too nlp.to_disk("en_core_web_sm_without_ner") …

Last idea:

Also, I would recommend using the textcat.correct recipe too. Don't worry about annotating as much as about getting a feel for how your model performs and where's its blind spots. Even better, correct any mistakes it's making and retrain.

If your current annotations are in textcat_data and your model is my_textcat_model, you can load that dataset as your source by prefixing dataset:

python -m prodigy textcat.correct correct_data my_textcat_model dataset:textcat_data ...

I think you may uncover some insight by correcting examples (and improve your model too!).

Let me know if you make progress!