Active learning and correct directly, instead of binary classification first

[didmar (Didier Marin)]

Hello,

I was wondering why there is no NER recipe that selects documents using active learning then lets the user correct the tags right away (like ner.correct does)?
What is the benefit of separating the process in two with ner.teach and ner.silver-to-gold?

(I have beta tested another labeling tool that would select a document through active learning, apply the model to predict all the tags, then let the user correct them. This seemed way more efficient to me, hence my confusion.)

[ryanwesslen]

Bonjour @didmar!

@didmar

I was wondering why there is no NER recipe that selects documents using active learning then lets the user correct the tags right away (like ner.correct does)?
What is the benefit of separating the process in two with ner.teach and ner.silver-to-gold?

Thanks for your question! I think you're asking why the active learning recipe frames tasks as binary, not like a manual recipe like ner.correct. This has come up before:

Why is there no active learning for manual/gold standard annotations?

You can definitely provide an update() callback function in a recipe that uses the manual interface. Custom recipes are very useful in general, as the Python API is much more powerful than the recipe arguments. Ultimately we decided to avoid trying to cram every combination of possible options into the built-in CLI. If you make the CLI too complicated, you end up programming via the CLI, and learning too many arbitrary details. At some point it’s better to switch over to Python. The source for …

As Matt mentioned, this is just a design choice to avoid cramming too many things within the built-in recipes.

There is nothing stopping you from developing your own custom recipe to do this. It's important to think of the built-in recipes as the floor of what's possible, not the ceiling of what's possible. The built-in recipes are there to get you started with smart defaults, but may need to be modified or extended.

One Prodigy pro tip: You can view the built-in recipes by finding your installed Prodigy package location (run prodigy stats and view Location:), and then find the recipes folder. For example, the ner recipes can be found inrecipes/ner.py. If you want, you can combine and modify the recipes to your preferences by using different sorters. We outline in the NER docs pseudo code to write a custom recipe with NER active learning.

Somewhat related, Matt mentioned in this post earlier some of the evolution in Prodigy's custom recipes design and we're we've had to rethink over time:

Can we bring back --seeds for textcat.teach?

@claycwardell As Ryan said, thanks for the feedback on this . I agree that we should get updates to the videos and tutorials, we're working on that. While I don't want to rule out that it's a simpler problem, here's a bit of context around why matching everything up with earlier versions isn't always easy. Machine learning techniques have continued to develop over the time we've been developing Prodigy, and we've obviously wanted to continue to keep things moving forward. However someti…

@didmar

(I have beta tested another labeling tool that would select a document through active learning, apply the model to predict all the tags, then let the user correct them. This seemed way more efficient to me, hence my confusion.)

Thank you for this feedback! I'm going to write up an internal ticket exploring more. If you have other feedback, please fill out our user survey. It's given us a treasure-trove of fixes and enhancements on top of our upcoming releases.

[didmar (Didier Marin)]

Hi @ryanwesslen !

Thank you for the detailed answer!
OK, I think I understand better now why you would want to split it in two, first focusing on find the most uncertain predictions without carefully going though the whole document.
In my case, ner.teach usually comes back multiple times to the same document for different spans, but I guess this is because my model is not that good yet?