Thanks for your answer, it makes everything clear for me now.
Now that I have some annotations with the custom interface, I'm having a bit of trouble reviewing them. Previously in this thread, we discussed the necessity of a custom review recipe for this.
I want to review the annotations for relations + spans (RE and NER) and the categories from the choice interface. Ideally, I want to review all of these annotations at once in the same review interface. One thing to note here is that in some cases, annotations for the same text (example) may have been presented with slightly different choice options because we're still coming up with the final set of these options. Therefore, I have updated the options on the go, so an annotation done on the same text after the update will have more options to choose from than a previous one. In the future, all annotations will have the same set of options. But for now, I wonder if it's possible to see the RE and NER annotations of each annotator, and just one choice interface showing which options were selected by each annotator. Essentially, this is like combining the review interfaces for relations and choice. How can I set up a custom recipe to accomplish this?
For now, I'm reviewing only the NER + RE annotations using the default review interface. However, the recipe was not grouping annotations with the same input hash and even task hash. This seems to be because the review recipe will create its own hashes, and these end up being different for the same examples (same text), perhaps because they have different values for the options or html fields. I managed to work around this by using a Python script to export a JSONL with the fields only relevant for the NER and RE tasks (text, tokens, spans, relations, _timestamp, etc). Now it does group annotations for the same text. So I have a question: Is it possible to make the review recipe group annotations just by text and not consider other fields like options or html when reviewing NER + RE with the relations interface?
As we have discussed previously, the review interface supports reviewing only oneview_id at a time, so it is not possible to change it from the recipe level. It's a UI (front-end) logic. There are reasons to this constraint, though. Notably, the review UI renders multiple versions of the same example, so if you stack up the multiple versions of multiple tasks the UI can become really hard to read / revise. I understand that for your purposes this may be irrelevant. For most cases, though, it is. For v2 of Prodigy we are planning more flexibility on the front end by allowing users to write their own front-end components but at the moment it is not possible and review permits one view_id only. Of course you can pass in as input a dataset with annotations from multiple view_ids but then you need to specify which view_id should be used for the review.
The usual procedure would be to review each task i.e. relations + span labels and choice task separately.
RE review internal hashing
That's right, the review recipe rehashes the examples with the relations view_id to make sure the right annotations are being compared. The keys used are:
Be ware that if you have different option sets these will be handled as different inputs. However they essentially are different inputs as the annotators who didn't have these options couldn't choose them, so there's not much point in treating them as exactly same inputs and comparing them as such, imo.
If you need to modify the way hashing is done, it should be enough to modify these keys in your local copy version of the review recipe (it's on line 31 and 31). You would be removing "html" and "options" from input.
As a reminder your local version of the recipe as at your Prodigy installation path that you can find under Location when you run prodigy stats.
I understand that the Prodigy review recipe can't display more than one view_id interface at the same time. Is it possible to build a custom recipe that can present more than one view_id type at the same time in a review like manner (displaying annotations for the same example from each annotator)? In our case, the choice annotations are very dependent on the spans and relations annotations, so it would be very beneficial to review them at the same time.
I searched this support forum but I couldn't find examples of people building custom review recipes that combine more than one view_id. I'm trying to assess the feasibility and complexity of building such a custom recipe. From your answer above, I understand it is not possible to use the default recipe for this, but I want to clarify if this is the case too when trying to build a custom review-like recipe.
I have split the the different topics in the original thread to separate threads with more representative labels to make it easier for other users to search for answers.
Perhaps I have not explained that well enough in my previous answers, but the constraint that it is currently impossible to render two review streams or a single review stream for two different UIs comes from the front-end code. Effectively, you would need to customize Prodigy front end to render it correctly which is currently impossible.
As I mentioned, we are planning to make the front end more flexible by allowing users to implement their own components but it is not possible just yet.
The only solution that occurs to me is to have relations review stream in one tab and the choice review stream the other in the other tab and merge the reviewed annotations in post-processing. You would have to click accept in both tabs, though to move forward which is inconvenient of course. I'm afraid there's not much to be done about from the recipe level.