In Mechanical Turk, there is a feature to prevent the annotators from gaming the labeling process, and the idea is to randomly present a gold standard question, where we already know what the correct answer is. If the annotator misses the gold standard question, then we give feedback and record that they missed the answer.
With Prodigy, do you have suggestions on how this can be implemented?
We were thinking about doing the following:
- configure the answer to a gold standard question in the meta field & use hide_meta in the prodigy.json
- at the end of the task, we log the number of failures. if this exceeds some threshold, we throw out the user’s results
- is it possible to prevent the user from moving forward until they have answered the gold standard correctly?
- is it possible to modify the meta field after they answer something? for example, it isn’t useful for us to know that they got the right answer eventually. instead, we want to know if they got the gold standard question wrong at least once.