training on a regression task

[LucySkywalker]

Hi there, I am looking to train a BERT model to estimate the level of friendliness in the text. The data is from online discussion fora. Typically only a sentence or two will be evaluated. The friendliness level should be assigned on a continuous scale (hence the: "problem_type": "regression"), for example:

• -3 for very unfriendly
• -2 for unfriendly
• -1 for somewhat unfriendly
• 0 for neutral
• 1 for somewhat friendly
• 2 for friendly
• 3 for very friendly

Every text will be evaluated by a number of annotators and the target value for the text will be an average of all responses. Thus a target value can be (for example) 2.3 for quite friendly text.

I think that I can get away with using using 7 categories for annotation of the above mentioned friendliness levels, however it gets more complicated if I would like to benefit from all features offered by prodigy like training model in a loop with picking for human evaluation of examples that the model is most "confused about" (prefer uncertain). However it seems that the output should not be regarded as a set of discrete (unordered) categories, but instead is should be regarded as a continuous variable.

For example, for discrete categories the error/difference between 1.9 and 2.1 is considered "larger" than between 1.1 and say 1.8 (as for the first example if we think in terms of discrete categories we change the category label, that is not the case for the second example as both values stay within the same category, however obviously 0.2 < 0.7).

I have trained proof of concept for this task using pyTorch, now I would love to add more data with the help of prodigy

Can textcat.manual be easily modified to be used with continuous output?

Would prodigy help in selection of the most difficult examples (prefer uncertain) for human evaluation?

Thank you for any help or suggestions.

[ryanwesslen]

hi @LucySkywalker!

Thanks for your questions and welcome to the Prodigy community

I would recommend creating a custom Prodigy recipe with your existing PyTorch workflow. The good news is there's a text classification template with docs that describe how to do this. I've tried to answer both of your questions directly below.

@LucySkywalker

Can textcat.manual be easily modified to be used with continuous output?

So there are two parts to the question: the UI/interface (i.e., creating a way to capture the continuous output) and the aligned spaCy component to train your model.

UI/Interface

It is possible to create a custom Prodigy interface that can allow a continuous annotation. This is where you can use the concept of blocks to combine different interfaces.

For example, you can create a slider like this:

is there a slider interface?

Hi! There's not currently a slider interface, no. If you want to use values on a scale, you could set them up as choice options. This also gives you keyboard shortcuts out-of-the-box, so you can just press 1 for 1, and so on. Alternatively, you can also easily add your own slider with the html interface and a JavaScript function that adds the selected value to the current task. Here's an example: <input type="range" id="slider" min="1" max="5" onChange="handleChange(this)" /> function handleC…

Here's a more detailed example of a slider:

saving user ratings to a task from html inputs

Sorry this is a bit uninutitive, but the prodigyanswer event is fired once the task is already answered, so you couldn't use that to update the current task. Instead, it's better to attach the updating to the onChange event of the slider, so the current task is updated whenever the slider changes. document.addEventListener('prodigymount', () => { // This runs once the page is set up document.querySelector('#score_slider').addEventListener('change', event => { window.prodigy.up…

The @tannonk also has a helpful GitHub repo with the recipes.

https://github.com/tannonk/prodigy_human_evaluation/tree/master/examples

There may be some HTML/JavaScript customizing that's needed but that can at least allow you to get users' input in a continuous format.

Training / Model

This is more of a challenge. Out-of-the-box, textcat.manual is for training model's using spaCy's textClassifier textcat or multilabel_textcat components. The problem is to my knowledge, neither of those components offer a regression training:

Training as a regression model · explosion/spaCy · Discussion #7007

Therefore, if you wanted to train your model with spaCy, you'd need to create a custom spaCy component to handle training of a continuous value (regression).

An alternative: use your existing PyTorch model/setup

Given you have a PyTorch PoC, I would recommend skipping spaCy and use Prodigy to create your own PyTorch workflow.

There's a section in the Text Classification documentation on how to create a custom Prodigy recipe for a different model workflow.

As linked in those docs, I'd recommend starting with this script:

explosion/prodigy-recipes/blob/master/textcat/textcat_custom_model.py

@LucySkywalker

Would prodigy help in selection of the most difficult examples (prefer uncertain) for human evaluation?

Yes! See the sub-section in the docs but you can use one of Prodigy's sorters to specify how you want Prodigy to use active learning to modify the order of your records for annotation:

from prodigy.components.sorters import prefer_uncertain

model = Model()
stream = model(stream)
stream = prefer_uncertain(stream)

After implementing this workflow, you'll likely still need to use the slider example above so that annotators can provide their continuous annotation value.

I hope this helps and let us know if you have questions (or please post back if you make progress!).

[LucySkywalker]

@ryanwesslen many thanks for your tips and suggestions! I'll be trying to implement them in the next few weeks, I'll post back on my progress...