Asking multiple questions in one task (different input types)

Hi!
I want to ask coders one mandatory and two optional questions in each task. Reading the docs, I would think there are two options, but I've been struggling to get where I want this to go for a few hours so I hope I can get some advice here. I basically want to show a piece of text and then ask three questions:

  1. mandatory: single choice from multiple (this would be prodigy's choice UI element)
  2. optional: simple yes / or no (in HTML terms, this would be a single checkbox)
  3. optional: free text field (this would be a text_input in prodigy's terms)

The two ways I found are:

  • put everything, i.e., task content and the questions as HTML form elements, in a prodigy HTML UI element. This seems to work, but it also looks like I have to deal with a lot of css formatting due to the css styles that are applied by prodigy by default. Also, how can I access what users type into a form element, e.g., a textarea?
  • use the prodigy's blocks UI element and use various elements. How would the corresponding "blocks" definition need to look like? I'm struggling with getting the second question, i.e., the single on/off switch (single checkbox) into this; I would probably use a choice element here (with a single, non-mandatory option) but since I already use one choice element for the first question, this seems to not work.

Could you please help me regarding my two questions? Also, I'd love to know which way you would choose. If there is a way regarding checking mandatory and optional - that would be great, but we can also deal with this by using proper labeling instructions.

Thanks a lot in advance!

Hi! That's an interesting question :slightly_smiling_face:

Since you can only have one instance of an interface per task (because the data would overwise get messy and have conflicts), I would probably do something like this:

blocks = [
    {"view_id": "choice"},
    {"view_id": "html", "html_template": HTML_TEMPLATE, "javascript": JAVASCRIPT},
    {"view_id": "text_input"}
]

For the choice block, your task could specify a list of "options" that's displayed. For the html block, you could have a simple HTML template with a checkbox and a small JavaScript script that listens to the checked event of the box and updates the task with whether it's checked. And the text_input will create a text box and add whatever the user types as the key "user_info" (or a custom field_id).

The HTML and JavaScript could look like this – when Prodigy is mounted, you listen to any change of the checkbox and then add a field "checked" to the task indicating whether it was checked (or whatever else you need):

<input type="checkbox" id="checkbox" />
document.addEventListener('prodigymount', () => {
    const checkbox = document.querySelector('#checkbox')
    checkbox.addEventListener('change', event => {
        window.prodigy.update({ checked: event.target.checked })
    })
})

In theory, that's possible with the same approach shown above. You can define your HTML elements, access them in JavaScript, listen to changes (or other things) and call window.prodigy.update to update the current annotation task.

At the moment, that's a bit tricky, but Prodigy v1.10 should have you covered :smiley: You'll then be able to define a validate_answer callback in your custom recipe that's called on every answer the annotator submits in the UI. So you can pretty much assert anything and/or raise errors that are then shown to the annotator in an alert. And the task can only be answered if validation passes (or it can be skipped, of course).

Here's an example for how you could validate a mandatory choice selection:

def validate_answer(eg):
    selected = eg.get("accept", [])
    if not selected:
        raise ValueError("Please select at least one option!")

Hi @ines,

Is there a way currently (even in a hacky way) to have a validate_answer functionality? So that the annotator cannot move to the next task until the annotated answers are validated?

Is there a timeline for release of Prodigy v1.10?

P.S. thank you for your wonderful support on this forum.

Thanks :slightly_smiling_face: I don't want to commit to an exact ETA yet but I definitely want to get v1.10 out within the next few weeks, the sooner the better. I'm currently working on a video that walks through some of the new features.

You could implement something similar on the client using custom JavaScript – I just can't immediately think of a good and easy way to actually prevent the annotator from moving on. So you'd have to tell them to undo and correct the annotation. Here's an example that listens to the prodigyanswer event and then shows an alert if the answered task is invalid:

document.addEventListener('prodigyanswer', event => {
    const { task, answer } = event.detail
    // Perform your checks here and show an alert
    if(answer != 'ignore' && task.accept.length == 0) {
        alert ('Select at least one option. Please undo and correct your annotation.')
    }
})

I guess if you really wanted to automatically take the user back and undo, you could try and simulate a click on the undo button? Haven't tried this yet and it might have unintended side-effects, but basically: document.querySelector('.prodigy-button-undo').click().

1 Like

Hi @ines , thanks for your timely reply!

Given your code, I got two follow-up questions:

  1. If the HTML checkbox is checked in one task, it will also be checked in the next tasks, so users will have to manually click or unclick it every time. I tried to unset it using some simple javascript (see recipe below), but this does not work. Perhaps because the javascript is executed only once when the webapp is started up? What would be a suitable spot do unset the checkbox? EDIT: It seems the js-code is never executed at all, I tried adding an alert("test"); as the first line, but the popup never shows up.
  2. The html's checkbox state is not stored in the results using the code you posted (see my full recipe below). The output is as follows: {"gid":"allsides_3871_1604_25_Brown_64_69","text":"Asked whether the encounter would have unfolded the same way if Brown had been white, Wilson said yes.","html":"Asked whether the encounter would have unfolded the same way if <span style=\"background-color:#beaed4;font-weight:bold;text-decoration:underline\">Brown had been white, Wilson said yes.","target":"Brown","options":[{"id":"q1_1","text":"strongly negative attitude (dislike)"},{"id":"q1_2","text":"2"},{"id":"q1_3","text":"3"},{"id":"q1_4","text":"neutral"},{"id":"q1_5","text":"5"},{"id":"q1_6","text":"6"},{"id":"q1_7","text":"strongly positive attitude (like)"}],"_input_hash":659129653,"_task_hash":-1447545197,"_session_id":"bias1-fabienne","_view_id":"blocks","accept":["q1_5"],"remarks":"remarki","answer":"accept"}

This is the recipe:

import prodigy
from prodigy.components.db import connect
from prodigy.components.loaders import JSONL


@prodigy.recipe(
    "newstsc",
    dataset=prodigy.recipe_args["dataset"],
    file_path=("Path to texts", "positional", None, str),
)
def sentiment(dataset, file_path):
    """Annotate the sentiment of texts."""
    stream = get_stream_loop(file_path, dataset)

    HTML_TEMPLATE = '<input type="checkbox" id="is_wcl_checkbox" /><label for="is_wcl_checkbox">Is bias by word choice and labeling regading highlighted subject (how it is described, not what)?</label>'
    JAVASCRIPT = """document.querySelector('#is_wcl_checkbox').checked = false;
    document.addEventListener('prodigymount', () => {
    const checkbox = document.querySelector('#is_wcl_checkbox')
    checkbox.addEventListener('change', event => {
        window.prodigy.update({ checked: event.target.checked })
    })
})
"""

    blocks = [
        {"view_id": "choice"},
        {"view_id": "html", "html_template": HTML_TEMPLATE, "javascript": JAVASCRIPT},
        {"view_id": "text_input", "field_id": "remarks", "field_label": "Remarks"},
    ]

    return {
        "dataset": dataset,  # save annotations in this dataset
        "stream": stream,
        "view_id": "blocks",  # use the choice interface
        "config": {
            "choice_auto_accept": False,
            "feed_overlap": True,  # Whether to send out each example once so it’s annotated by someone (false) or whether to send out each example to every session (true, default). Should be used with custom user sessions set via the app (via /?session=user_name).
            "force_stream_order": True,  # Always send out tasks in the same order and re-send them until they’re answered, even if the app is refreshed in the browser
            "instructions": "/prodigy/manual.html",
            "blocks": blocks,
        },
    }


def get_stream_loop(file_path, dataset):
    # to prevent that no tasks are shown even though there are still unlabeled tasks
    # left, cf.
    # https://support.prodi.gy/t/struggling-to-create-a-multiple-choice-image-classification/1345/2
    db = connect()
    while True:
        stream = get_stream(file_path)
        hashes_in_dataset = db.get_task_hashes(dataset)
        yielded = False
        for eg in stream:
            # Only send out task if its hash isn't in the dataset yet, which should mean
            # that we will not have duplicates
            if eg["_task_hash"] not in hashes_in_dataset:
                yield eg
                yielded = True
        if not yielded:
            break


def get_stream(file_path):
    stream = JSONL(file_path)  # load in the JSONL file
    stream = add_options(stream)  # add options to each task

    for eg in stream:
        eg = prodigy.set_hashes(eg)
        yield eg


def add_options(stream):
    """Helper function to add options to every task in a stream."""
    options = [
        {"id": "q1_1", "text": "strongly negative attitude (dislike)"},
        {"id": "q1_2", "text": "2"},
        {"id": "q1_3", "text": "3"},
        {"id": "q1_4", "text": "neutral"},
        {"id": "q1_5", "text": "5"},
        {"id": "q1_6", "text": "6"},
        {"id": "q1_7", "text": "strongly positive attitude (like)"},
    ]
    for task in stream:
        task["options"] = options
        yield task

Thanks a lot for your help! :slight_smile:

PS: I removed some parts of an earlier version of this post as I was able to fix the issue by one modification of my recipe, see this message's history, in case someone is interested - to fix the described issue, I added one line of code that I commented out earlier, which added the options to the stream.

Hi! Just tried to run your code and got it to work fine :slightly_smiling_face: The only problems were:

  1. The "javascript" currently can't be overwritten by the block, so you have to add it in the "config". So basically, it didn't run before, which is why nothing happened. (Also make sure you don't have any "javascript" value in your prodigy.json btw, otherwise you'd override the recipe config.)
  2. There's no need to run document.querySelector('#is_wcl_checkbox').checked = false at the beginning here – this would only run once anyways, before Prodigy is mounted, and result in an error (because the checkbox isn't there yet). If you want to make sure that the checkbox is rerendered every time a new task comes in, the easiest way is to add an HTML attribute that changes for every task – e.g. by adding a property like key that's populated by a variable unique to the task. For example, key="{{_task_hash}}" uses the task hash, which we know is always unique.
HTML_TEMPLATE = '<input key="{{_task_hash}}" type="checkbox" id="is_wcl_checkbox" /><label for="is_wcl_checkbox">Is bias by word choice and labeling regading highlighted subject (how it is described, not what)?</label>'
1 Like

Hi @ines, thanks for the update! Using your updates, I got the overall process working, but only the results of the first task in a session contain the html-based checkbox information. Subsequent task miss the checkbox states entirely, no matter if I check and/or uncheck them again or leave them as they are (unchecked). Could you let me know how to add html-based results, e.g., in my case the checkbox states, to all tasks?

I don't have time to test it at the moment but try moving the change handler into a function and the onChange attribute of the checkbox – like onChange="handleChange()". And then read out the value in a handleChange function. I think what might be happening here is that resetting and rerendering the checkbox also removes the event listener, so you want to make sure you're listening to the right checkbox on each render.

Thanks a lot! I ended up with an inline function in the html form fields, since when invoking a function that I defined in the JAVASCRIPT variable, I got errors stating that the function was not defined (yet). Not sure why that is, but maybe because the JAVASCRIPT content is inserted into the webpage after all the HTML is rendered and processed?

For anyone interested in this solution, a minimalistical HTML in my recipe looks like this:

<input key="{{_task_hash}}" type="checkbox" id="checkbox_is_sentence_issue" onchange="(function(checkbox_object){window.prodigy.update({ [checkbox_object.id]: checkbox_object.checked })})(this)" />
<label for="checkbox_is_sentence_issue">Sentence fragment or multiple sentences</label>

Ah cool, thanks for the update – that actually looks pretty elegant! :+1:

Ah, that's weird – because by the time your HTML is rendered and you manage to click on something, the JS should definitely be available. Maybe something else went wrong there, but anyway, looks like you found a solution :slightly_smiling_face:

1 Like