# Bug when using choice and review interface **URL:** \ **Category:** Uncategorized\ **Tags:** textcat, solved, review\ **Created:** [March 2, 2020, 2:11pm UTC](https://support.prodi.gy/t/bug-when-using-choice-and-review-interface/2609 "2020-03-02T14:11:31Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Kairine](https://avatars.discourse-cdn.com/v4/letter/k/ed8c4c/32.png) [@Kairine](https://support.prodi.gy/u/Kairine)\ **Post date:** [March 2, 2020, 2:11pm UTC](https://support.prodi.gy/t/bug-when-using-choice-and-review-interface/2609/1 "2020-03-02T14:11:31Z")
Hello, I am using Prodigy 1.9.7, and i'm trying to manually annotate 50 texts with 8 different labels. Somehow I keep running into this message: ![bug](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/a/a01a86374b9d6972af2ac092eddb601ac69d418d.jpeg) Here are all the tests I made: First I went with the textcat.manual recipe: `prodigy textcat.manual new_50 data/new_50.json --label labels.txt`. There wasn't any error message, but the interface shows **No tasks available**. Then I tried with one label to see if there was something wrong with the input file: `prodigy textcat.manual mail_new_50 data/mail_new_50.json --label NO_NEW`. This time it worked. Then I tried using the mark recipe as an alternative: `prodigy mark mail_new_50 data/mail_new_50.json --label label/labels.txt --view-id choice` That's where I first had the **Something went wrong** message. And again I tried using one single label: `prodigy mark mail_new_50 data/mail_new_50.json --label NO_NEW --view-id classification` And again it worked. I noticed that every time I used the choice interface there was a problem; not sure if that's actually the reason tho. I went on to create a custom recipe which I got the inspiration from [this post on multi-class textcat with patterns](https://support.prodi.gy/t/textcat-manual-with-patterns-argument/2039) (thanks a lot to all contributors btw !). A few slight changes were made to adapt it to my project, but the general operation was the same. The recipe worked well and was successfully used in a multi-annotator campaign. So I obtained several annotation sets, and I wanted to review those annotations. First I called the review recipe this way: `prodigy review mail_new_gold mail_new_50` This lead to an error **No None found in the example** followed by the suggestion of adding `--view-id` option. Which I thought was odd because I used Prodigy 1.9.7 for the whole project and therefore, as I understand it, no need to specify an interface. But I added the option anyway: `prodigy review mail_new_gold mail_new_50 --view-id choice` Where for the second time I ran into the **something went wrong** message. And finally I decided to specify session names instead of dataset name: `prodigy review mail_new_gold mail_new_50-danrun,mail_new_50-danielle --view-id choice` This time it worked for a while (13 examples reviewed) before the **something went wrong** message came up. The same .json and .txt files have been used with other recipes and it worked pretty well, so I'm guessing maybe something's wrong with my local installation ? Btw it was installed on a docker container, I don't know if this could be a contributing fact. Do you have an idea how to fix this ? Thanks in advance ! 🙂 ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [March 2, 2020, 2:51pm UTC](https://support.prodi.gy/t/bug-when-using-choice-and-review-interface/2609/2 "2020-03-02T14:51:49Z")
Hi! I think there are several things going on here. First, do you have validation enabled (`"validate": true`)? Because some of these problems should have been caught by the data validation, which should give you clearer feedback about what the underlying problem is, before starting the server 🤔 > [@Kairine](#): > > `prodigy mark mail_new_50 data/mail_new_50.json --label label/labels.txt --view-id choice` > That's where I first had the **Something went wrong** message The problem here is that the `mark` recipe will render _exactly_ what comes in with the data, using a given interface. The [`choice` interface](https://prodi.gy/docs/api-interfaces#choice) expects an incoming task to have one or more `"options"`. If those are not present, Prodigy can't render the data. > [@Kairine](#): > > First I called the review recipe this way: > `prodigy review mail_new_gold mail_new_50` > This lead to an error **No None found in the example** followed by the suggestion of adding `--view-id` option. Where does the `mail_new_50` data come from and how was it created? And which interface was used? Somehow the dataset ended up with an example without a `"_view_id"` value. Prodigy will add this automatically, so it's definitely strange that this happened. Did you ever import data to this dataset from a file etc.? > [@Kairine](#): > > `prodigy review mail_new_gold mail_new_50 --view-id choice` > Where for the second time I ran into the **something went wrong** message. And finally I decided to specify session names instead of dataset name: > `prodigy review mail_new_gold mail_new_50-danrun,mail_new_50-danielle --view-id choice` > This time it worked for a while (13 examples reviewed) before the **something went wrong** message came up. This sounds like you might have ended up with a mix of data in different formats created with different interfaces in one or more of the datasets. The `choice` interface expects the examples to have `"options"` and a list of `"accept": []`, containing the selected options. If you're trying to review datasets where some examples have options and others don't, that's not going to work. An easy way to test this would be to write a script that load each dataset and then loops over the tasks and prints/logs examples that do not have `"options"` or do not have a `"_view_id"` set. Then you can figure out where they come from. > [@Kairine](#): > > First I went with the textcat.manual recipe: > `prodigy textcat.manual new_50 data/new_50.json --label labels.txt` . > There wasn't any error message, but the interface shows **No tasks available**. What's in your `new_50` dataset? The most likely explanation here is that the dataset already contains annotations for the texts in `new_50.json`. By default, Prodigy will skip examples that are already annotated, so you're not asked the same question twice. If the dataset already contains annotations with multiple-choice options for the given texts, but none yet for text plus single `"label"`, that'd also explain why running `textcat.manual` with only one label showed you questions to annotate. ---
**Author:** ![Kairine](https://avatars.discourse-cdn.com/v4/letter/k/ed8c4c/32.png) [@Kairine](https://support.prodi.gy/u/Kairine)\ **Post date:** [March 2, 2020, 4:13pm UTC](https://support.prodi.gy/t/bug-when-using-choice-and-review-interface/2609/3 "2020-03-02T16:13:42Z")
> [@ines](#): > > Hi! I think there are several things going on here. First, do you have validation enabled (`"validate": true`)? Because some of these problems should have been caught by the data validation, which should give you clearer feedback about what the underlying problem is, before starting the server 🤔 I checked prodigy.json, the validation is indeed enabled. Strange 🤔 > [@ines](#): > > > [@Kairine](#): > > > > `prodigy mark mail_new_50 data/mail_new_50.json --label label/labels.txt --view-id choice` > > That's where I first had the **Something went wrong** message > > The problem here is that the `mark` recipe will render _exactly_ what comes in with the data, using a given interface. The [`choice` interface](https://prodi.gy/docs/api-interfaces#choice) expects an incoming task to have one or more `"options"`. If those are not present, Prodigy can't render the data. I tried adding options to the original data file and it worked ! Thanks ! > [@ines](#): > > > [@Kairine](#): > > > > First I called the review recipe this way: > > `prodigy review mail_new_gold mail_new_50` > > This lead to an error **No None found in the example** followed by the suggestion of adding `--view-id` option. > > Where does the `mail_new_50` data come from and how was it created? And which interface was used? Somehow the dataset ended up with an example without a `"_view_id"` value. Prodigy will add this automatically, so it's definitely strange that this happened. Did you ever import data to this dataset from a file etc.? > > > [@Kairine](#): > > > > `prodigy review mail_new_gold mail_new_50 --view-id choice` > > Where for the second time I ran into the **something went wrong** message. And finally I decided to specify session names instead of dataset name: > > `prodigy review mail_new_gold mail_new_50-danrun,mail_new_50-danielle --view-id choice` > > This time it worked for a while (13 examples reviewed) before the **something went wrong** message came up. > > This sounds like you might have ended up with a mix of data in different formats created with different interfaces in one or more of the datasets. The `choice` interface expects the examples to have `"options"` and a list of `"accept": []`, containing the selected options. If you're trying to review datasets where some examples have options and others don't, that's not going to work. The dataset is initialized with `mail_new_50.json`, in which I have 50 texts without other properties. It is then annotated by several people using `choice` interface. I checked every one's session and every entry has a `_view_id` value. The only entries without a `_view_id` are the originals. That explains why it worked when I specified which sessions to review, since the original texts are no longer reviewed. But I still don't understand why it worked for 13 examples then stopped ? They all have `"options"` and `"accept"`. > [@ines](#): > > > [@Kairine](#): > > > > First I went with the textcat.manual recipe: > > `prodigy textcat.manual new_50 data/new_50.json --label labels.txt` . > > There wasn't any error message, but the interface shows **No tasks available**. > > What's in your `new_50` dataset? The most likely explanation here is that the dataset already contains annotations for the texts in `new_50.json`. By default, Prodigy will skip examples that are already annotated, so you're not asked the same question twice. > > If the dataset already contains annotations with multiple-choice options for the given texts, but none yet for text plus single `"label"`, that'd also explain why running `textcat.manual` with only one label showed you questions to annotate. It's actually `mail_new_50` and `mail_new_50.json`, I copied the wrong command. Sorry about that. And by adding `"options"` to the original it worked too ! Just to be sure tho: basically I added all options to the texts, and in the cmd I still need to specify labels, is this expected ? ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [March 3, 2020, 10:09am UTC](https://support.prodi.gy/t/bug-when-using-choice-and-review-interface/2609/4 "2020-03-03T10:09:49Z")
Thanks for checking – this definitely makes sense then 🙂 I'll check if we can add more data validation here, because everything related to tasks having incorrect data formats is stuff we can easily catch early. > [@Kairine](#): > > The dataset is initialized with `mail_new_50.json` , in which I have 50 texts without other properties. You don't have to upload any data to your dataset before annotating – the datasets are only for the collected annotations and to provide the input data, just set the argument on the command line. If you pre-import the raw data, it's going to be treated as annotations and used during training, the review process etc., which is typically not what you want. > [@Kairine](#): > > That explains why it worked when I specified which sessions to review, since the original texts are no longer reviewed. But I still don't understand why it worked for 13 examples then stopped I think what might have happened is that example 1 to 13 were annotations created with Prodigy and the `choice` interface. Then came example 14, which was one of the raw unannotated examples without `"options"` that you had imported before, so the interface couldn't render it. > [@Kairine](#): > > Just to be sure tho: basically I added all options to the texts, and in the cmd I still need to specify labels, is this expected ? `textcat.manual` will take care of adding `"options"` to each incoming example, based on the `--labels` you set on the command line. So you don't have to do that in your original data. I think the problem in your case was that the dataset _also_ contained raw unannotated examples, which were the ones that ended up causing all the problems. ---
**Author:** ![Kairine](https://avatars.discourse-cdn.com/v4/letter/k/ed8c4c/32.png) [@Kairine](https://support.prodi.gy/u/Kairine)\ **Post date:** [March 3, 2020, 1:21pm UTC](https://support.prodi.gy/t/bug-when-using-choice-and-review-interface/2609/5 "2020-03-03T13:21:46Z")
OK it all makes sense now. Thank you so much !