When using the prodigy.review recipe, which examples does it show on Prodigy?
Trying to enumerate…
answer agreeanswer disagreeanswer agree, but span doesn’t (ner and/or ner_manual)answer is IGNORE
Hi – sorry if this wasn’t fully clear. There are essentially two scenarios:
Here, we consider examples identical and part of the same review example if their _task_hash is identical. We consider different answers disagreements. For instance, you can collect multiple annotations about different entities in the same text and annotate accept / reject. In review mode, you’ll see each of those examples and the “accept” vs. “reject” decisions on them. “ignore” answers will always be filtered out.
Here, we consider examples identical and part of the same review example if their _input_hash is identical. We consider different annotations (i.e. _task_hash values) disagreements. For instance, if annotations on the same text highlight different entities, you’ll see one example in review mode with all different variations below. The most common version (i.e. the one that has the most number of annotations), will be pre-highlighted. We do not merge anything together here – we consider each individual annotations its own version. “ignore” and “reject” answers will always be filtered out.
In general, the review interface should always show you all examples (e.g. if the answer was accept/reject for binary annotations, or accept for manual annotations). The goal is to give you a final corrected dataset with only one final version of each example that you can then train from etc. The different annotation versions from across the sessions are all preserved in a "versions" key btw – so you’ll always be able to trace the annotations back to the different versions they were based on.