Feed results of choice to next choice

1

Hi.
I'm trying to use Prodi.gy to capture expert feedback on a GenAI PoC. The users need to A) pick the best of several responses and B) provide feedback on the quality of that response.

I'm using choice with custom options per task for A. Ideally the user would immediately be presented with the feedback (B) choices, but I can't put two choices in blocks.

  1. Is there a way to get the feedback on the same task/screen? It's really disruptive for these users to have to go through the set twice every day during this PoC.
  2. Is there a way to feed the answer from choice into the input of another job? I can see how to use the entire dataset in a reannotation but that's not the task here. If I have to run a bunch of scripts between the completion of A and start of B, my expert users won't be able to complete both tasks in their busy days.

Thanks.

2

Hi there!

I can't put two choices in blocks.

Unfortunately that's correct. It's explained in the accordion at the bottom of this section of the docs. Multiple blocks of the same type are only supported for the text_input and html interfaces.

Is there a way to get the feedback on the same task/screen? It's really disruptive for these users to have to go through the set twice every day during this PoC.

One avenue that might be worth considering is to use another annotation interface. Might the text input work for you here? You can use field suggestions but also allow folks to add more context that way.

Is there a way to feed the answer from choice into the input of another job?

This should be possible with some custom Python code. Assuming that you have data that looks a bit like this:

{
  "text": "Pick the odd one out.",
  "options": [
    {"id": "BANANA", "text": "🍌 banana"},
    {"id": "BROCCOLI", "text": "🥦 broccoli"},
    {"id": "TOMATO", "text": "🍅 tomato"}
  ],
  "accept": ["BROCCOLI"],
  "answer": "accept"
  "_input_hash": 123
}

You could do something like:

import srsly
from prodigy.components.db import connect 

db = connect()

examples = db.get_dataset_examples("name_of_dataset")

def new_examples(examples):
    for ex in examples:
        choice_id = ex['accept'][0]
        yield {
            "text": ex['options'][choice_id]['text'], 
            "orig": ex["_input_hash"]
        }

srsly.write_jsonl("new_examples_path.jsonl", new_examples(examples))

This writes the texts into a new file with a link to the original input hash of the original example. Would that work?