Hi Team Prodigy,
We had some questions about the correct way to customize Prodigy's active learning features for our use case.
In order to minimize annotation time, we tried to create an NER recipe that combines ner.teach
and ner.correct
. At a high-level, our goal is to present the annotator with abstracts that are tagged by the model, let the annotator update the annotations using the ner.correct/ner.manual
interface, and then use active learning to update the model similar to how it is done in ner.teach
. In other words, we want to use active learning, but instead of the binary annotation interface used in ner.teach
, we want to use the annotation interface in ner.manual/ner.correct
, because we believe it is possibly a more efficient use of the annotator's time.
The annotated examples are passed onto either the update
function of the NER model that is being retrained or the update
function returned by combine_models
function that combines the NER model with the PatternMatcher
.
Our questions are:
- What is the format of input arguments that we should pass onto the
update
functions above in order to ensure that the models are updated in the best possible way? Could you provide us with the function signature as well as either the function code or a sufficient explanation of it? - For example, is it ok if we just pass to the update functions
accept
s but neverreject
s? Is it ok if we pass a list of text and associated metadata where each dictionary in the list itself includes multiple spans? Should we transform this input in any way in order to use theupdate
function correctly? Maybe we should transform the inputs into a binary format ofaccept
andreject
and make sure each example has exactly one span? We have a more detailed example of what we pass as input below. - Is there a way to empirically check whether the model is being updated correctly? Two quantitative values we observe are the loss returned by the
update
function as well as a logging statement which says things likePROGRESS: Estimating progress of 0.1667
. Currently, we do not understand how the loss and progress values are related to each other. Can you help us understand that, and more generally, help us find a quantitative way of verifying whether we are using active learning successfully? - Does the number of examples passed onto the
update
function matter? What are best practices in regards to that?
Currently, we pass to the update
function a list, where each element of the list has the keys text
, spans
, and an answer
. spans
itself is a list where each element is a dictionary as follows:
{'start': 74,
'end': 83,
'token_start': 13,
'token_end': 13,
'label': 'PROTEIN'}
thank you for your help,
berk