Reviewing coreference annotations

Hello! I have a custom recipe to create coreference annotations in Danish, and am now working on a script to compare annotations from three different annotators. It almost works with the built-in review recipe (when I specify view-id to be relations), but it is only showing one of the annotation versions and not any conflicts between the files, both in regards to noun phrase spans and coreference annotations. How are the files merged (I've set is_manual = True)? Would it be possible to set different colors for differing coref annotations and spans, for example?

Hi! Which version of Prodigy are you using and if you're not on the latest (v1.10.8), can you try updgrading? I'm pretty sure a recent update should resolve the problem, which was essentially caused by the new relations UI not being considered a "manual" interface :slightly_smiling_face: (also see this thread).

I was still using the version 1.10.7, so yes this helps. Thank you! However, the merged annotations still seems to be one of the versions I'm comparing. I tried changing the labels in the other document to "COREF2" from "COREF" with different colors to make both of them appear, but it didn't help. So, would there be another way to show conflicts in the merged file? That would make comparing them a lot easier!

By merged, do you mean the main editable version displayed at the top? Prodigy won't attempt to merge both versions because it may not always be possible due to conflicts – and in a lot of cases, it's not what you want because many conflicts are caused by one annotator making a mistake. So you wouldn't want that to be added automatically. By default, Prodigy will pick the version most sessions (e.g. annotators) agreed on and if there are multiple with the same frequency, the first one.

Yes, that's what I meant :slight_smile: And yes, that makes sense, but in our case, we have max 3 annotators per document, so I was just wondering if there would be an easier way to spot where their annotations differ.