Custom model Requirements

custom
usage

#1

I would like to use my own custom model (not a spacy model. e.g. pytorch, tensorflow, keras) with prodigy interface in active learning. What are the requirements of the model and how to integrate into custom recipe?


(Ines Montani) #2

To make any model integrate with Prodigy’s active learning workflow, you mainly need to expose two functions:

  • a predict function that takes an iterable stream of examples in Prodigy’s JSON format, scores them and yields (score, example) tuples
  • an update callback that takes a list of annotated examples and updates the model accordingly

Here’s a pseudocode example of how this could look in a custom text classification recipe. How you implement the individual components of course depends on the specifics of your model.

import copy
from prodigy.components.loaders import JSONL
from prodigy.components.sorters import prefer_uncertain

@prodigy.recipe('custom')
def custom_recipe(dataset, source):
    stream = JSONL(source)
    model = load_your_model()

    def predict(stream):
        for eg in stream:
            predictions = get_predictions_from_model(eg)
            for label, score in predictions:
                example = copy.deepcopy(eg)
                example['label'] = label
                yield (score, example)

    def update(answers):
        for eg in answers:
            if eg['answer'] == 'accept':
                update_model_with_accept(eg)
            elif eg['answer'] == 'reject':
                update_model_with_reject(eg)
        loss = get_loss()
        return loss

    return {
        'dataset': dataset,
        'view_id': 'classification',
        'stream': prefer_uncertain(predict(stream)),
        'update': update
    }

You can also find more details on the expected formats and component APIs in your PRODIGY_README.html or in the custom recipes workflow.


Using Stanford CoreNLP during active learning
#3

Hi,

Thanks for your instruction. I used this method to apply my customized pytorch model, which make loss converged when I do batch training. But when I use this algorithm to teach annotation, the result is close to even worse than samples which is random chosen from whole dataset. The experiment I took is to predict the sentiment of the IMDB review is positive or negative. The experiment group is using the annotations generated by prodigy active learning process while the baseline group is a random order chosen from whole dataset. For both groups, I trained successive data based on the model trained on previous samples .
image

I’m very confused about the result, it seems algorithm that is good for supervised learning is not good for active learning. I’m curious if there is any requirement for the customized model output? The meaning of my customized score is the probability of positiveness now. Could you tell me the logic of the ```prefer_uncertain`` function? I just think it’s important to know it well to build more suitable model.


(Matthew Honnibal) #5

Just so I’m understanding your experiment correctly: what’s step in the graph above? Is it the number of data samples? If so, is this training from only one epoch? What happens if you train for multiple epochs?

My more general answer: it’s true that active learning isn’t a good fit for every problem. The IMDB sentiment corpus was designed to investigate particular text classification techniques, so the dataset has several characteristics that make models converge well on the data. Specifically:

  • The texts are quite long.
  • The texts are of similarish length.
  • Exactly two classes.
  • Perfect class balance.
  • Few boundary cases.
  • Low annotation noise.

These problem characteristics make the dataset a relatively bad example for active learning, I think. The class composition is especially relevant. In many datasets you’ll have a lot of labels, with one label making up a lot of the examples, some classes that are rare but easy to predict (because the examples are all very similar), and some other classes that are easily confused.

Another thing to consider is that when doing active learning, it can matter a lot how fast your model responds to new examples. The default text classification model for Prodigy actually has a few features designed to make it learn better under an active learning regime. The most important feature is that it’s an ensemble of a unigram bag-of-words model and a CNN. During the first 10-20 weight updates, the CNN is still performing at close to chance, but the unigram bag-of-words model can already have learned a lot, because it starts off with such a useful inductive bias about the problem.

So, it’s possible that your model architecture learns a bit too slowly, and that’s one reason why your active learning might not perform well. But it might not be the decisive reason — I do think IMDB is a very tough example for active learning, so I wouldn’t be surprised if Prodigy’s default configuration actually doesn’t beat the baseline on it either. I haven’t run that experiment; I’d be interested to find out the result.


(Akshita Sood) #6

Hi Ines,
I wanted to know that if there are multiple predictions for one eg, then we will get that example multiple times for annotation ? If not, how will the score be calculated for that particular example ?

Thanks