Hi Prodigy Team,
Context
I'm working on a custom recipe for evaluation of long text generation outputs (multiple sentences) and would like to collect continuous preference ratings from multiple models.
My current approach is to display the source text with two model outputs side-by-side and an HTML range input below that allows the user to simply indicate the degree to which they prefer a particular output. Ideally, these pair-wise preference scores would be combined using TrueSkill to then then rate multiple systems.
So far I've been using an EventListener on prodigyanswer to try to capture the value from the range input when the user accepts/rejects the task.
Problem
The problem I'm facing is how to save the value from the range input to the database along with the task.
What I've Tried
-
I was expecting to be able to update the value by simply using
event.detail.task.score = user_score, but this fails to update the score value completely. -
After searching for solutions on the forum, I came across
window.prodigy.update(). Doing something likewindow.prodigy.update({ score: user_score })almost works but gives me an off-by-one error, saving the score with the following task.
Minimal Example
Below is a minimal example of my current recipe containing the two approaches I've tried so far (custom_javascript1 and custom_javascript2).
import prodigy
from prodigy.components.loaders import JSONL
custom_css = """
/* make the annotation window wider for side-by-side display */
.prodigy-container {
max-width: 200em;
white-space: normal;
}
/* reduce the font size for all annotation elements */
.prodigy-content * { font-size: 14px; }
/* set the two model outputs for comparison side-by-side */
.prodigy-content:nth-child(3) {
display: grid;
grid-template-columns: 1fr 1fr;
grid-gap: 100px;
}
/* style settings for custom slider */
.slidecontainer {
margin: auto;
width: 80%; /* Width of the outside container */
}
.slider {
-webkit-appearance: none; /* Override default look */
width: 100%; /* Set a specific slider handle width */
height: 25px; /* Slider handle height */
background: #d3d3d3; /* slider background */
outline: none; /* Remove outline */
opacity: 0.7; /* Set transparency (for mouse-over effects on hover) */
-webkit-transition: .2s; /* 0.2 seconds transition on hover */
transition: opacity .2s;
cursor: pointer; /* Cursor on hover */
}
.slider:hover {
opacity: 1; /* Fully shown on mouse-over */
}
"""
slider_html = """
<div class="slidecontainer">
<input type="range" list="tickmarks" min="-100"
max="100" class="slider" id="score_slider" step=1>
<datalist id="tickmarks">
<option value="-100" label="100%"></option>
<option value="-90"></option>
<option value="-80"></option>
<option value="-70"></option>
<option value="-60"></option>
<option value="-50" label="50%"></option>
<option value="-40"></option>
<option value="-30"></option>
<option value="-20"></option>
<option value="-10"></option>
<option value="0" label="0%"></option>
<option value="10"></option>
<option value="20"></option>
<option value="30"></option>
<option value="40"></option>
<option value="50" label="50%"></option>
<option value="60"></option>
<option value="70"></option>
<option value="80"></option>
<option value="90"></option>
<option value="100" label="100%"></option>
</datalist>
</div>
"""
custom_javascript1 = """
document.addEventListener("prodigyanswer", event => {
var slider = document.getElementById("score_slider");
// save the value to the annotation task
user_score = parseFloat(slider.value);
event.detail.task.score = user_score;
console.log('User score =', user_score);
// reset the slider to default value
slider.value = slider.defaultValue;
})
"""
custom_javascript2 = """
document.addEventListener("prodigyanswer", event => {
var slider = document.getElementById("score_slider");
// save the value to the annotation task
user_score = parseFloat(slider.value);
window.prodigy.update({ score: user_score });
console.log('User score =', user_score);
// reset the slider to default value
slider.value = slider.defaultValue;
})
"""
@prodigy.recipe(
"preference_slider",
dataset=("The dataset to use", "positional", None, str),
source=("The source data as a JSONL file", "positional", None, str),
)
def choice(dataset: str, source: str):
"""
Rating pairwise model outputs with a preference slider
"""
# stream in lines from JSONL file yielding a
# dictionary for each example in the data.
stream = JSONL(source)
return {
"view_id": "blocks",
"dataset": dataset, # Name of dataset to save annotations
"stream": stream, # Incoming stream of examples
"config": {
"blocks": [
{"view_id": "html", "html_template": "<p>{{text}}</p>"},
{"view_id": "html", "html_template": "<div>{{options.a.text}}</div><div>{{options.b.text}}</div>"},
{"view_id": "html", "html_template": slider_html},
],
"global_css": custom_css,
"javascript": custom_javascript2
},
}
Example data:
{"text":"This is the first long source text.","id":01,"options":{"a":{"id":"model_a","text":"This is model a output."},"b":{"id":"model_b","text":"This is model b output."}},"score":0}
{"text":"This is the second long source text.","id":02,"options":{"a":{"id":"model_c","text":"This is model c output."},"b":{"id":"model_b","text":"This is model b output."}},"score":0}
{"text":"This is the third long source text.","id":03,"options":{"a":{"id":"model_d","text":"This is model d output."},"b":{"id":"model_c","text":"This is model c output."}},"score":0}
{"text":"This is the fourth long source text.","id":04,"options":{"a":{"id":"model_b","text":"This is model b output."},"b":{"id":"model_d","text":"This is model d output."}},"score":0}
Question
Could you provide me with some insights for how to best save the selected values from an arbitrary HTML input?
Thanks!