hi @nlp-guy!
Thanks for your post and I apologize for the delay on getting back to you.
While we don't have detailed documentation on a case study (it's something we've been thinking about), there are several links that list out ideas on how to approach this problem.
First, we have a custom prodigy-recipe for question-and-answering. Assuming you have input candidate q-a, it allows users to "accept"
, "reject"
, or "ignore"
each question-answer pair.
It assumes the source data (qa pairs) are in this jsonl
format:
{"question": "What color is the sky?", "question_answer": "blue"}
Also the compare
recipe could work if you wanted to frame the task as an A/B evaluation (e.g., provide the question, provide the QA model's answer and a dummy answer and allow the user to choose the "best answer").
If you're interested in something where the user has to provide a free form response (e.g., answer a question), text_input
recipe is one option.
If the prodigy-recipe for question-and-answering doesn't meet your need, here's a post that has an alternative.
It shows how to use blocks
and JavaScript for a custom extractive QA recipe (details in the ):
Let us know if you have any further questions!