Customise Prodigy interface for NLP Q&A Task with Multiple Questions

hi @nlp-guy!

Thanks for your post and I apologize for the delay on getting back to you.

While we don't have detailed documentation on a case study (it's something we've been thinking about), there are several links that list out ideas on how to approach this problem.

First, we have a custom prodigy-recipe for question-and-answering. Assuming you have input candidate q-a, it allows users to "accept", "reject", or "ignore" each question-answer pair.

It assumes the source data (qa pairs) are in this jsonl format:

{"question": "What color is the sky?", "question_answer": "blue"}

Also the compare recipe could work if you wanted to frame the task as an A/B evaluation (e.g., provide the question, provide the QA model's answer and a dummy answer and allow the user to choose the "best answer").

If you're interested in something where the user has to provide a free form response (e.g., answer a question), text_input recipe is one option.

If the prodigy-recipe for question-and-answering doesn't meet your need, here's a post that has an alternative.

It shows how to use blocks and JavaScript for a custom extractive QA recipe (details in the :thread:):

image

Let us know if you have any further questions!