Call task variables/show scores in choice config

, , ,
2

Hi! In the upcoming version (currently available via the nightly pre-relase), Prodigy supports an additional "meta" field on the individual "options" that will be displayed next to the options. So if you're using the nightly, you can format the task like this:

{
    "text": "Some text",
    "options": [
        {"id": "BANANA", "text": "BANANA", "meta": "0.7"},
        {"id": "APPLE", "text": "APPLE", "meta": "0.2"},
        {"id": "APPLE", "text": "ORANGE", "meta": "0.6"}
    ]
}

This is also how the new textcat.correct recipe does it under the hood. It uses a threshold to decide which options to pre-select based on their scores, and those labels are automatically added as the "accept" key. So with the default threshold of 0.5, you'd end up with "accept": ["BANANA", "APPLE"].

If you're using Prodigy v1.10 that doesn't yet support the "meta" field in the options, you could achieve the same by just adding the scores in the "text" field of the option directly:

def add_model_predictions_to_stream(stream):
    for eg in stream:
        # Let's assume this returns a list of (label, score) tuples
        results = get_some_predictions(eg["text"])
        options = [{"id": label, "text": f"{label} (score: {score})"} for label, score in results]
        selected = [label for label, score in results if score >= 0.5]
        eg["options"] = options
        eg["accept"] = selected
        yield eg

If you want to style the score in a different way (e.g. make it smaller, use a monospace font etc.) you could also provide a key "html" instead of "text".

1 Like