Using Prodigy for A/B testing ML models

1

Hi there!

My team and I are considering using Prodigy for the internal A/B testing of our trained recommendation models. The system takes in a set of ~30 questions and recommends ~10 questions. The experts will be asked to mark any questions that are “relevant” to the context questions. We were wondering if we can set Prodigy up to -

  1. Show long-form text as context, possibly with a scroll for the “input” box
  2. Allow users to select multiple checkboxes, with the ability to select/de-select all the recommended questions

We intend to use the “best” performing model from this A/B testing into production. Hope to hear soon! Any feedback on better ways to frame this problem if we were to use Prodigy is also appreciated!

Best,
Akshay

2

I'm not 100% sure what you mean by "scroll for the input box", but you can stream in

You could also use the "spans" to highlight a sentence within a longer text visually. However, if the question requires lots of context, I'd recommend to keep the cognitive load low and etiher ask one question at a time, or reduce the length of the text. Ideally, you want the annotators to give feedback in small steps and as quickly as possible.

Sure, that should be pretty easy to do using the choice interface and "choice_style": "multiple". The data that goes in usually looks like this:

{
    "text": "Some text",
    "options": [
        {"id": 0, "text": "Option one"},
        {"id": 1, "text": "Option two"},
        {"id": 2, "text": "Option three"}
    ]
}

When the user annotates the text, an "accept" key will be added containing the selected IDs – for example "accept": [0, 2]. However, you can also pre-populate these in the data you stream in – so if you have 10 options and your system "selects" 0 and 2, you can add "accept": [0, 2] to the incoming data and the options will be pre-selected. The annotator can then correct them if necessary.

For an example of how to write a custom recipe like this, check out the custom recipes workflow or the code in our prodigy-recipes repo: