Arriving at consensus on documents

We have multiple users and multiple documents, and would like the same document to be annotated by multiple users so that we can take a consensus approach.

For example, say there are options A,B,C,D for a document, we'd like to see that the responses from 6 respondents were ABAADA. Then we can mark the consensus response as A.

Please could you help me know if this sort of thing is possible, and if so, how?
Many thanks in advance, Charlie

1 Like

Plus one on this! Being able to spot consensus is very important to an application I am building. Is there / will there be support for this?

Hi! I think what you're looking for is the review workflow: https://prodi.gy/docs/recipes#review

It lets you load data from one or more datasets, including all annotations from different named sessions, merge all annotations on the same input automatically and create a final annotation with the final consensus, that you can then train from. For manual annotation tasks, Prodigy will also try and pre-select the answers that most annotators agreed with. The underlying JSON data will preserve all different annotations that led to the final decision, so you'll always be able to reconstruct what lead to the final annotation.

Here's an example of the interface for a multiple choice text classification task: