# Review recipe - how to review? **URL:** \ **Category:** Uncategorized\ **Tags:** review\ **Created:** [May 22, 2024, 9:08pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239 "2024-05-22T21:08:28Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![tlingren](https://avatars.discourse-cdn.com/v4/letter/t/c2a13f/32.png) [@tlingren](https://support.prodi.gy/u/tlingren)\ **Post date:** [May 22, 2024, 9:08pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/1 "2024-05-22T21:08:28Z")
I have annotations completed by two annotators, and trying to review them. I tried the following command, but was unable to see the annotations in the interface: > screen -r python3 -m prodigy review Med\_EM\_Problem\_detection\_dataset\_v23.jsonl Med\_test\_schema\_v0.4 Error: > The running recipe is configured for multiple annotators using named > sessions with feed\_overlap=True, or via a task router setting, but a client is > requesting questions using the default session. For this recipe, open the app > with ?session=name added to the URL or set feed\_overlap to False in your > configuration. I received the error. I understand feed\_overlap is set because we want annotators to double annotate. How do I maintain that ability and still review? What should the url look like ?session=review or something else? There is nothing to see if I don't have a session id set. ---
**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\ **Post date:** [May 23, 2024, 6:24pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/2 "2024-05-23T18:24:38Z")
Hi @tlingren, The typical workflow for the `review` recipe is to have just one person to the revision - a single final adjudicator of the annotation conflicts. This is the recommended way to ensure that the conflicts are being resolved consistently. Would you like to have different reviewers review the same or different examples? In any case with `feed_overlap` set to True, Prodigy expects to be accessed with so called "named session" and the `name` in `?session=name` pattern should be name (or an identifier) of the reviewer e.g. `sam` so the full URL could be `http://localhost:8080/?session=sam` . This name or identifier will be used to uniquely identify the annotations done by this reviewer (it will be stored in the `_session_id`and `_annotator_id` attributes of the task and a copy of all annotations will also be stored in a separate dataset with this identifier in the name). If you don't provide it, and the `feed_overlap` is set to True, Prodigy will error out. ---
**Author:** ![tlingren](https://avatars.discourse-cdn.com/v4/letter/t/c2a13f/32.png) [@tlingren](https://support.prodi.gy/u/tlingren)\ **Post date:** [May 23, 2024, 7:11pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/3 "2024-05-23T19:11:54Z")
Thanks for the reply. To clarify, I can have one of the annotators set as the 'adjudicator' and turn feed\_overlap to true, which when I run the review recipe will display all annotator's output overlayed on the same text? Is feed\_overlap a command line parameter or do I need to edit a py code? and which one? ---
**Author:** ![tlingren](https://avatars.discourse-cdn.com/v4/letter/t/c2a13f/32.png) [@tlingren](https://support.prodi.gy/u/tlingren)\ **Post date:** [May 24, 2024, 2:51pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/4 "2024-05-24T14:51:45Z")
I have created a prodigy.json in the main home directory and added {feed\_overlap=True} That eliminates the error message, however, I'm getting three boxes of annotations (one presumably for the master, and one for each of the 2 annotators). Could this be because we are using both spans and rel in the annotation? I can make rel decisions here, but see the output of spans when looking at the individual annotators output. ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/7/7f24210704f0ccbfc6cba1fbd3d20e4773f15644.png) ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/d/d73be35c9ec9e2a593474fccd39d0de39cdabd0e.png) ---
**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\ **Post date:** [May 25, 2024, 11:48am UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/5 "2024-05-25T11:48:30Z")
Hi @tlingren, Yes, you can have one adjudicator with `feed_overlap` set to `true` you just need to access Prodigy server with the "named session" pattern we talked about before. `feed_overlap` can be set in the global Prodigy config file `prodigy.json` (by default located in your home\_directory/.prodigy) or local Prodigy config file located in your current working directory (just like you did). For completeness, you can also set it via CLI or [environment variables](https://prodi.gy/docs/install#config-overrides). The settings from all these sources are merged so it's just a matter of convenience where you set it. Btw. prodigy.json is like a python dictionary in terms of format so for the setting to take the effect it should be: ```python {"feed_overlap": true} ``` The spelling errors won't raise en error, it would be silently ignored instead. (You can find more details on what else can be set via config [here](https://prodi.gy/docs/install#config)) Now, as for the the `review` interface: you're right, it will have the "adjudicated" or "master" view and then it will show alternative annotations from different annotators. The adjudicator can select using the majority vote (which is what will show up as default), select a different alternative (by clicking on the alternative) or manually edit the "adjudicated" view. So the number of boxes will depend on how many alternatives there are (not on a particular "view id"). You can also automatically accept the ones that all annotators agreed on with `-A` flag. Do you want to adjudicate both relation annotations and span annotations? If so, you should also add `"relations_span_labels": []`to your `prodigy.json` where you would list the target span labels in the square brackets (this is just a workaround, we are planning to support `--span-labels` argument in `review` but we currently don't which why it's not documented). Or the problem is that you're not seeing the relation annotation in the "annotator boxes"? 🤔 It might be that this particular example did not receive any relation annotation? You'd need to double check with the source jsonl file as a first check. ---
**Author:** ![tlingren](https://avatars.discourse-cdn.com/v4/letter/t/c2a13f/32.png) [@tlingren](https://support.prodi.gy/u/tlingren)\ **Post date:** [May 28, 2024, 1:20pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/6 "2024-05-28T13:20:42Z")
In this annotation we are using --labels and --span\_labels . The --labels are relationships. The --span\_labels are entities What the issue was that above, it said "no\_label", but there was clearly at least one annotation (for span\_labels) in the transcript seen (second image). If we have to do 2 separate reviews (one for relations and one for spans, that's ok). But I did the above (relationship labels show up, but only the spans show up in the annotation. And the wrap button does not work with these multiple windows. It only seems to work for the top box. It looks like I can check it in the other boxes, but nothing happens and the screen refreshes. And if the top box is wrapped, it will unwrap. If I click on an annotated text, the screen refreshes and it jumps to the front of the text. It's a frustrating UI to try to work through. ---
**Author:** ![tlingren](https://avatars.discourse-cdn.com/v4/letter/t/c2a13f/32.png) [@tlingren](https://support.prodi.gy/u/tlingren)\ **Post date:** [May 29, 2024, 2:55pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/7 "2024-05-29T14:55:08Z")
@magdaaniol I completed the review session, and saved each snippet. But when I exported the db , the annotations in the jsonl file were the same. > python3 -m prodigy db-out Med\_test\_schema\_v0.4 \> ./med\_test\_v04\_consensus Is there a different way to export a review session ? There wasn't anything in the documentation about a review export. ---
**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\ **Post date:** [May 30, 2024, 1:44pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/8 "2024-05-30T13:44:43Z")
Hi @tlingren, Sorry to hear your experience with the `review` was not exactly smooth. I admit that there are a few rough edges to polish for reviewing the `relations` UI and the fact the UI `wrap` checkbox doesn't work is one of them. It's definitely on our TODO list to fix it. In the meantime, just to summarize the the workarounds to the issues you've reported: 1. The lack of span labels in the `review` UI You can add them via prodigy config by specifying ```python "relations_span_labels": ["PER","ORG","LOC"] # example list of labels to add ``` So there's no need to to do the revision of spans and relations in two passes. 1. Dysfunctional `wrap` checkbox in `review` Again, this setting can be effectively added via prodigy config by specifying ```python "wrap_relations": true ``` These are all settings of the `relations`UI documented [here](https://prodi.gy/docs/api-interfaces#relations-settings). The `review` API does not handle them all yet, but they can be definitively added via the config file. Again, we are aware of these shortcomings and will fix it as soon as possible. As to your last question about the export: there's nothing specific about exporting the review dataset. is `Med_test_Schema_v04` the name of the dataset that you used as `dataset` parameter to the `review` recipe? In other words, when calling the `review` recipe, you used a commad that looked something like: ```python python3 -m prodigy review my_reviewed_dataset annotated_dataset --label X,Y ``` What you want to export with `db-out` is `my_reviewed_dataset`. Each example will contain the copies of the particular versions under the `versions` keys but the top level annotation is the adjudicated one. You can also check the `_session_id` should correspond to the adjudicator session\_id (either timestamp or name if a named revision session was used) ---
**Author:** ![tlingren](https://avatars.discourse-cdn.com/v4/letter/t/c2a13f/32.png) [@tlingren](https://support.prodi.gy/u/tlingren)\ **Post date:** [May 30, 2024, 1:54pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/9 "2024-05-30T13:54:09Z")
> [@magdaaniol](#): > > `"wrap_relations": true` Thank you for the detailed response. WRT the db-out, there was no difference between the original and reviewed. So perhaps my review command was incorrect. What I did was: my\_reviewed\_dataset = original source file annotated\_dataset = DATASET (as listed in the file below) ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/2/212f532ad03f4ebd2c08482f61ceab6743a51f1c.png) I specified the labels in the .prodigy/.prodigy config and not the command line. ---
**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\ **Post date:** [May 30, 2024, 4:36pm UTC](https://support.prodi.gy/t/review-recipe-how-to-review/7239/10 "2024-05-30T16:36:37Z")
Right, just to be clear: my\_reviewed\_dataset (in the example command I provided) should be a fresh empty dataset where to store your revisions. And that's the one you should db-out. If you used an existing dataset as the target of the `review` your revisions would be stored together with what is already in there. I think the most efficient way here would probably be for you to revise the `review` command you used against the documented`review` API [here](https://prodi.gy/docs/recipes#review), where all parameters are defined. That hopefully clears out any doubts about the meaning of each dataset name.