We are trying to review entries which have been annotated (via textcat) by multiple annotators. We have questions over how Prodigy is handling text entries when an annotator says to ignore it. Our impression is that Prodigy will never show a text entry once it's labeled 'Ignore' by at least one annotator.
Is that right? And if so, is there a way to change that behavior? Thanks!
Hi! By default, the
review recipe will exclude all ignored answers, yes - it shouldn't ignore an example if another annotator answered something else, but it wouldn't show the ignored answer.
I definitely see the point, though, that if you use the "ignore" action to indicate "don't know the answer", you may still want to see who skipped it. So we should probably at least expose an option that lets you toggle whether to show who ignored that example.
In the meantime, you should also be able to change this yourself pretty easily: you can run
prodigy stats to find the location of your Prodigy installation. Then open
recipes/review.py and find the following line (should be around line 102) and remove it:
examples = (eg for eg in examples if eg["answer"] != "ignore")
Sorry for never responding back! For some reason, I didn't get a reply notification at all and I've been stomping out other issues, I forgot about this entire conversation. Thanks for the information, that is great to know, and we'll give it a spin!
Update: Just released v1.10.5, which introduces a
--show-skipped flag in the
review recipe that will show examples that would otherwise be skipped (ignored answers or rejected annotations in manual mode).
Hey! I found this thread while Googling around - not sure if there's a better way to do this but I haven't been able to find it yet: I'd like to be able to review only the ignored answers. We made up this workflow of, a person labels, and if they are not sure, you ignore it and someone more knowledgable will review only the ignored ones. I'm solving this now by running a python script that filters out the answers that are ignored - it's just a bit of a hassle with having to first export the data with db-out, then filter it with the script, then do db-in, review, db-out, and then db-merge (I think!).