Hi, I would like to ask for advice on the review recipe.
I'm at the beginning of a large annotation project (20.000 examples) with a team of annotators. We plan to annotate using ner.manual and annotate each example twice by using a group_a and group_b session ID.
After an example has been annotated by both groups, I run the review recipe.
- Is it possible to auto-accept (and add to review dataset) all annotations that were annotated identically by the two groups? That would save clicking through the 90% of examples where that was the case.
- I'll review throughout the process. Is it possible to let review only show the examples that already have 2 annotations (not just 1)?
What is the easiest way to do that?
I checked the custom recipes Github repo and read the review.py file, but am a bit lost with how I would go about modifying the recipe.
Any hints would be much appreciated