Reviewing Ignored Cases

Hi all!

We are trying to review entries which have been annotated (via textcat) by multiple annotators. We have questions over how Prodigy is handling text entries when an annotator says to ignore it. Our impression is that Prodigy will never show a text entry once it's labeled 'Ignore' by at least one annotator.

Is that right? And if so, is there a way to change that behavior? Thanks!

1 Like

Hi! By default, the review recipe will exclude all ignored answers, yes - it shouldn't ignore an example if another annotator answered something else, but it wouldn't show the ignored answer.

I definitely see the point, though, that if you use the "ignore" action to indicate "don't know the answer", you may still want to see who skipped it. So we should probably at least expose an option that lets you toggle whether to show who ignored that example.

In the meantime, you should also be able to change this yourself pretty easily: you can run prodigy stats to find the location of your Prodigy installation. Then open recipes/ and find the following line (should be around line 102) and remove it:

examples = (eg for eg in examples if eg["answer"] != "ignore")

Sorry for never responding back! For some reason, I didn't get a reply notification at all and I've been stomping out other issues, I forgot about this entire conversation. Thanks for the information, that is great to know, and we'll give it a spin!

Update: Just released v1.10.5, which introduces a --show-skipped flag in the review recipe that will show examples that would otherwise be skipped (ignored answers or rejected annotations in manual mode).