training on a regression task

[ryanwesslen]

hi @LucySkywalker!

Thanks for your questions and welcome to the Prodigy community

I would recommend creating a custom Prodigy recipe with your existing PyTorch workflow. The good news is there's a text classification template with docs that describe how to do this. I've tried to answer both of your questions directly below.

@LucySkywalker

Can textcat.manual be easily modified to be used with continuous output?

So there are two parts to the question: the UI/interface (i.e., creating a way to capture the continuous output) and the aligned spaCy component to train your model.

UI/Interface

It is possible to create a custom Prodigy interface that can allow a continuous annotation. This is where you can use the concept of blocks to combine different interfaces.

For example, you can create a slider like this:

is there a slider interface?

Hi! There's not currently a slider interface, no. If you want to use values on a scale, you could set them up as choice options. This also gives you keyboard shortcuts out-of-the-box, so you can just press 1 for 1, and so on. Alternatively, you can also easily add your own slider with the html interface and a JavaScript function that adds the selected value to the current task. Here's an example: <input type="range" id="slider" min="1" max="5" onChange="handleChange(this)" /> function handleC…

Here's a more detailed example of a slider:

saving user ratings to a task from html inputs

Sorry this is a bit uninutitive, but the prodigyanswer event is fired once the task is already answered, so you couldn't use that to update the current task. Instead, it's better to attach the updating to the onChange event of the slider, so the current task is updated whenever the slider changes. document.addEventListener('prodigymount', () => { // This runs once the page is set up document.querySelector('#score_slider').addEventListener('change', event => { window.prodigy.up…

The @tannonk also has a helpful GitHub repo with the recipes.

https://github.com/tannonk/prodigy_human_evaluation/tree/master/examples

There may be some HTML/JavaScript customizing that's needed but that can at least allow you to get users' input in a continuous format.

Training / Model

This is more of a challenge. Out-of-the-box, textcat.manual is for training model's using spaCy's textClassifier textcat or multilabel_textcat components. The problem is to my knowledge, neither of those components offer a regression training:

Training as a regression model · explosion/spaCy · Discussion #7007

Therefore, if you wanted to train your model with spaCy, you'd need to create a custom spaCy component to handle training of a continuous value (regression).

An alternative: use your existing PyTorch model/setup

Given you have a PyTorch PoC, I would recommend skipping spaCy and use Prodigy to create your own PyTorch workflow.

There's a section in the Text Classification documentation on how to create a custom Prodigy recipe for a different model workflow.

As linked in those docs, I'd recommend starting with this script:

explosion/prodigy-recipes/blob/master/textcat/textcat_custom_model.py

@LucySkywalker

Would prodigy help in selection of the most difficult examples (prefer uncertain) for human evaluation?

Yes! See the sub-section in the docs but you can use one of Prodigy's sorters to specify how you want Prodigy to use active learning to modify the order of your records for annotation:

from prodigy.components.sorters import prefer_uncertain

model = Model()
stream = model(stream)
stream = prefer_uncertain(stream)

After implementing this workflow, you'll likely still need to use the slider example above so that annotators can provide their continuous annotation value.

I hope this helps and let us know if you have questions (or please post back if you make progress!).