Hi there!
My annotation task is for coders to verify prediction. I will present a paragraph and a predicted output that I would like the coders to select if they (1) Agree (2) Reject (3) Unsure as a single-choice option. Then, if the user select (2) Reject or (3) Unsure, I'd like to have a multiple choice options so they could select their reasons. I'd like the multiple choice selection to have some existing options but also free text input, and ideally, once a coder add something in the free text input, it will show up as a choice in the next annotation (and across sessions as well). Currently, I have implemented the text input block with field suggestions. However, it seems like once a coder select an option, the option list would not appear again for them to select additional options. Any help/suggestions would be highly appreciated! thanks!
Welcome to the forum @andreawwenyi ,
Could you see if this solution here would work for you?
Let me know if you need help adapting it to your use case!
As for modifying the available options set based on user answers:
There are two challenges related to it: 1) you might quickly end up with an unmanageable list of options due to coders introducing too many variations (even spelling mistakes) 2) since the mechanism of building the task is a function shared by all annotators one annotator's additions would affect what other annotators see unless you keep their sessions completely independent which would require a custom session factory and splitting tasks between the annotators upfront rather than using the Prodigy task router.
Hi @magdaaniol, thank you for the suggestion! The solution you pointed to generally match with what I am thinking of. However, I wonder if it's possible to make the following adaptations:
- I would like the resulting output to also document the first level option (e.g. "positive" or "negative"). The current solutions only document the second level options (i.e. {'selected': ['slow', 'wrong information']}). Is it possible to have something like {'selected': {"negative": ['slow', 'wrong information']} or {"selected": "negative", "selected_reasons": ["slow", "wrong information"]}?
- In the first level, I would like to add another option called "I'm not sure". There will be no other options under "I don't know" -- so user can just select "I don't know" and then go to the next task. The resulting output could be something like {"selected": {"I'm not sure": []}} or {"selected": "I'm not sure", "selected_reasons": []}?
- Among the first level options ("positive", "negative", "I'm not sure"), the user can only select one of them.
Is it possible to achieve this? Thanks again!!
Hi @andreawwenyi !
It is certainly possible to adapt this solution to meet the requirements you list. It will require expanding the html template and adding some new js functions to handle the new button actions and to store the data.
1. Adding the extra button:
In the template.jinja2, I've added a new dunno button on the last line. I've also added a few new js functions that should be triggered upon clicking i.e.: update_button to store the data from both "layers" of annotation and changeColor to visually signal that the "I'm not sure" button was clicked by turning it gray.
<button onclick="update_button('accept');toggle('accept')">Accept</button>
<div id="accept" style="display: none;">
<form style="display: block;">
{%- for reason in options["accept"] -%}
<input type="checkbox" class="checkbox" id="{{reason}}" name="{{reason}}" onchange="update()" style="margin: 0.4rem;"><label for="{{reason}}">{{reason}}</label><br>
{%- endfor -%}
</form>
</div>
<button onclick="update_button('reject');toggle('reject')">Reject</button>
<div id="reject" style="display: none;">
<form style="display: block;">
{%- for reason in options["reject"] -%}
<input type="checkbox" class="checkbox" id="{{reason}}" name="{{reason}}" onchange="update()" style="margin: 0.4rem;"><label for="{{reason}}">{{reason}}</label><br>
{%- endfor -%}
</form>
</div>
<button id="dunno" onclick="update_button('dunno');toggle('dunno');changeColor('dunno')">I'm not sure</button>
(I've also updated the IDs from the original example so that the whole snippet makes more sense.)
2. Updated custom js, with the definitions of new and updated functions:
function toggle(id) {
var x = document.getElementById(id);
if (id == "accept"){
reset("reject")
resetColor("dunno")
}else if (id == "reject"){
reset("accept")
resetColor("dunno")
} else if (id == "dunno"){
reset("accept")
reset("reject")
// clean up all the reasons that might have been stored
update()
}
if (id == "accept" || id == "reject"){
if (x.style.display === "none") {
x.style.display = "block";
} else {
x.style.display = "none";
}
}
}
function reset(id){
var x = document.getElementById(id);
x.style.display = "none"
var checkboxes = document.getElementsByClassName("checkbox");
for(let elem in checkboxes){
checkboxes[elem].checked = false;
}
}
function resetColor(id){
var x = document.getElementById(id);
x.style.backgroundColor = "#583fcf"
}
function update(){
// store the selected options under the `selected_reasons` key
var checkboxes = document.getElementsByClassName("checkbox");
var results = [];
for(let elem in checkboxes){
if(checkboxes[elem].checked){
results.push(checkboxes[elem].id)
}
}
prodigy.update({
selected_reasons: results
})
}
function update_button(id){
// store the selected button under the `selected` key
let selected;
if(id == "accept"){
selected = "Accept";
} else if (id == "reject"){
selected = "Reject";
} else if (id == "dunno"){
selected = "Unsure";
}
prodigy.update({
selected: selected
})
}
function changeColor(id) {
var element = document.getElementById(id);
if (element.style.backgroundColor === "rgb(128, 128, 128)") {
// If the button is already clicked, restore the original styles
element.style.backgroundColor = "#583fcf"
} else {
// If the button is not clicked, change the styles
element.style.backgroundColor = "#808080";
}
}
document.addEventListener('prodigyanswer', event => {
reset("accept")
reset("reject")
resetColor("dunno")
})
3. And finally the updated recipe. Not much going on there, just the mentioned updates to variable names:
import jinja2
import prodigy
from typing import Union
from pathlib import Path
from prodigy.util import msg
from prodigy import set_hashes
from prodigy.components.loaders import JSONL
def load_template(path: Union[str, Path]) -> jinja2.Template:
if not isinstance(path, Path):
path = Path(path)
if not path.suffix == ".jinja2":
msg.fail(
"Must supply jinja2 file.",
exits=1,
)
with path.open("r", encoding="utf8") as file_:
text = file_.read()
return jinja2.Template(text, undefined=jinja2.DebugUndefined)
@prodigy.recipe(
"super-custom-ui",
dataset=("The dataset to use", "positional", None, str),
source=("The source data as a JSONL file", "positional", None, str),
)
def user_post_interest_likert(
dataset: str,
source: str,
):
stream = JSONL(source)
options = {
"accept": [
"polite",
"fast",
"useful"
],
"reject": [
"slow",
"unfriendly",
"wrong information"
]
}
template = load_template("template.jinja2")
def add_template(stream):
for ex in stream:
ex['html'] = template.render(options=options)
yield set_hashes(ex)
custom_js = Path("custom.js").read_text()
def before_db(examples):
for ex in examples:
del ex['html']
print(ex)
return examples
return {
"view_id": "blocks",
"dataset": dataset, # Name of dataset to save annotations
"stream": add_template(stream), # Incoming stream of examples
"config": {
"blocks": [
{"view_id": "text"},
{"view_id": "html"},
],
"javascript": custom_js,
},
"before_db": before_db
}
When you run it the result should be like this:<
Since clicking one button, resets the state of the other buttons, the user, effectively can only choose one top layer button. After choosing the button, they will still have to accept the annotation via Prodigy buttons to move on to the next question.
The data should be stored in DB like so:
{'text': 'Uber’s Lesson: Silicon Valley’s Start-Up Machine Needs Fixing', 'meta': {'source': 'The New York Times'}, '_input_hash': 1726217019, '_task_hash': 963802771, '_view_id': 'blocks', 'selected': 'Accept', 'selected_reasons': ['polite'], 'answer': 'accept', '_timestamp': 1713975729, '_annotator_id': '2024-04-24_18-21-53', '_session_id': '2024-04-24_18-21-53'}
{'text': 'Pearl Automation, Founded by Apple Veterans, Shuts Down', 'meta': {'source': 'The New York Times'}, '_input_hash': 1124492168, '_task_hash': 1434540737, '_view_id': 'blocks', 'selected': 'Reject', 'selected_reasons': ['slow'], 'answer': 'accept', '_timestamp': 1713975735, '_annotator_id': '2024-04-24_18-21-53', '_session_id': '2024-04-24_18-21-53'}
{'text': 'How Silicon Valley Pushed Coding Into American Classrooms', 'meta': {'source': 'The New York Times'}, '_input_hash': -1116508957, '_task_hash': 1262922291, '_view_id': 'blocks', 'selected': 'Unsure', 'answer': 'accept', '_timestamp': 1713975737, '_annotator_id': '2024-04-24_18-21-53', '_session_id': '2024-04-24_18-21-53'}