Call task variables/show scores in choice config

, , ,
1

Hello,

I am curious whether it is possible to dynamically include arbitrary task variables in the 'choice' config? Specifically, I would like to display the confidence scores next to my label text like so:

(x) BANANA, score: 0.7
( ) APPLE, score: 0.2
(x) ORANGE, score: 0.6

I am actually using a custom recipe, a modified version of textcat.correct used to evaluate BigQuery results containing JSON that includes the model output...I currently have my labels come from command line argument but could just as easily parse them from the task 'meta' field, and since this is also where the scores are located I'm curious whether it is possible to include them in the 'choice' UI. My understanding is that there are only certain keys that are recognized on 'return' as you are feeding the stream rather than individual elements at that point, but not sure what I could name it in the task in order to call it here.

2

Hi! In the upcoming version (currently available via the nightly pre-relase), Prodigy supports an additional "meta" field on the individual "options" that will be displayed next to the options. So if you're using the nightly, you can format the task like this:

{
    "text": "Some text",
    "options": [
        {"id": "BANANA", "text": "BANANA", "meta": "0.7"},
        {"id": "APPLE", "text": "APPLE", "meta": "0.2"},
        {"id": "APPLE", "text": "ORANGE", "meta": "0.6"}
    ]
}

This is also how the new textcat.correct recipe does it under the hood. It uses a threshold to decide which options to pre-select based on their scores, and those labels are automatically added as the "accept" key. So with the default threshold of 0.5, you'd end up with "accept": ["BANANA", "APPLE"].

If you're using Prodigy v1.10 that doesn't yet support the "meta" field in the options, you could achieve the same by just adding the scores in the "text" field of the option directly:

def add_model_predictions_to_stream(stream):
    for eg in stream:
        # Let's assume this returns a list of (label, score) tuples
        results = get_some_predictions(eg["text"])
        options = [{"id": label, "text": f"{label} (score: {score})"} for label, score in results]
        selected = [label for label, score in results if score >= 0.5]
        eg["options"] = options
        eg["accept"] = selected
        yield eg

If you want to style the score in a different way (e.g. make it smaller, use a monospace font etc.) you could also provide a key "html" instead of "text".

1 Like
3

This is exactly what I needed, thanks so much!

1 Like