I'm attempting to follow the tutorial for Models as Annotators but when I get to the review phase, the annotations are not showing properly in the interface.
I ran 100 rows through a custom Spacey model and a GPT3.5 pipeline I assembled using the tutorial.
Here's a couple of jsonl example rows:
{"text": "Guitar toaster", "_input_hash": -2117718285, "_task_hash": -385783721, "options": [{"id": "3d_render", "text": "3d_render", "meta": "0.00"}, {"id": "abstract_art", "text": "abstract_art", "meta": "0.00"}, {"id": "anime", "text": "anime", "meta": "0.00"}, {"id": "architecture", "text": "architecture", "meta": "0.00"}, {"id": "character_design", "text": "character_design", "meta": "0.00"}, {"id": "cinematic", "text": "cinematic", "meta": "0.00"}, {"id": "conceptual_art", "text": "conceptual_art", "meta": "0.00"}, {"id": "dark_fantasy", "text": "dark_fantasy", "meta": "0.00"}, {"id": "fantasy_realism", "text": "fantasy_realism", "meta": "0.00"}, {"id": "fashion", "text": "fashion", "meta": "0.00"}, {"id": "graffiti", "text": "graffiti", "meta": "0.00"}, {"id": "illustration", "text": "illustration", "meta": "0.00"}, {"id": "interior_design", "text": "interior_design", "meta": "0.00"}, {"id": "logo", "text": "logo", "meta": "0.00"}, {"id": "painting", "text": "painting", "meta": "0.00"}, {"id": "photography", "text": "photography", "meta": "0.00"}, {"id": "portrait_photography", "text": "portrait_photography", "meta": "0.00"}, {"id": "poster", "text": "poster", "meta": "0.00"}, {"id": "product", "text": "product", "meta": "0.00"}, {"id": "sticker", "text": "sticker", "meta": "0.00"}, {"id": "surrealism", "text": "surrealism", "meta": "0.00"}, {"id": "typography", "text": "typography", "meta": "0.00"}, {"id": "ukiyo-e", "text": "ukiyo-e", "meta": "0.00"}, {"id": "vector_design", "text": "vector_design", "meta": "0.00"}, {"id": "watercolor", "text": "watercolor", "meta": "0.00"}, {"id": "wildlife_photography", "text": "wildlife_photography", "meta": "0.00"}], "accept": [], "tokens": [{"text": "Guitar", "start": 0, "end": 6, "id": 0, "ws": true}, {"text": "toaster", "start": 7, "end": 14, "id": 1, "ws": false}], "_annotator_id": "gpt35_turbo", "_session_id": "gpt35_turbo", "answer": "accept"}
{"text": "Guitar toaster", "_input_hash": -2117718285, "_task_hash": -385783721, "options": [{"id": "3d_render", "text": "3d_render", "meta": "0.13"}, {"id": "portrait_photography", "text": "portrait_photography", "meta": "0.17"}, {"id": "dark_fantasy", "text": "dark_fantasy", "meta": "0.11"}, {"id": "painting", "text": "painting", "meta": "0.13"}, {"id": "typography", "text": "typography", "meta": "0.11"}, {"id": "illustration", "text": "illustration", "meta": "0.13"}, {"id": "fantasy_realism", "text": "fantasy_realism", "meta": "0.11"}, {"id": "surrealism", "text": "surrealism", "meta": "0.09"}, {"id": "logo", "text": "logo", "meta": "0.11"}, {"id": "poster", "text": "poster", "meta": "0.12"}, {"id": "fashion", "text": "fashion", "meta": "0.09"}, {"id": "sticker", "text": "sticker", "meta": "0.11"}, {"id": "product", "text": "product", "meta": "0.55"}, {"id": "cinematic", "text": "cinematic", "meta": "0.11"}, {"id": "watercolor", "text": "watercolor", "meta": "0.09"}, {"id": "character_design", "text": "character_design", "meta": "0.11"}, {"id": "vector_design", "text": "vector_design", "meta": "0.11"}, {"id": "wildlife_photography", "text": "wildlife_photography", "meta": "0.12"}, {"id": "architecture", "text": "architecture", "meta": "0.10"}, {"id": "interior_design", "text": "interior_design", "meta": "0.09"}, {"id": "abstract_art", "text": "abstract_art", "meta": "0.15"}, {"id": "graffiti", "text": "graffiti", "meta": "0.11"}, {"id": "photography", "text": "photography", "meta": "0.13"}, {"id": "conceptual_art", "text": "conceptual_art", "meta": "0.12"}, {"id": "anime", "text": "anime", "meta": "0.11"}, {"id": "ukiyo-e", "text": "ukiyo-e", "meta": "0.09"}], "accept": ["product"], "tokens": [{"text": "Guitar", "start": 0, "end": 6, "id": 0, "ws": true}, {"text": "toaster", "start": 7, "end": 14, "id": 1, "ws": false}], "_annotator_id": "spacy_model", "_session_id": "spacy_model", "answer": "accept"}
{"text": "matte painting,fantasy forest of glowing trees and glowing flying insects, a lake reflecting the lights, myst on the air, cinematic feel, hyper realistic, high detail, blender render .", "_input_hash": -2110223808, "_task_hash": 813863613, "options": [{"id": "3d_render", "text": "3d_render", "meta": "1.00"}, {"id": "abstract_art", "text": "abstract_art", "meta": "0.00"}, {"id": "anime", "text": "anime", "meta": "0.00"}, {"id": "architecture", "text": "architecture", "meta": "0.00"}, {"id": "character_design", "text": "character_design", "meta": "0.00"}, {"id": "cinematic", "text": "cinematic", "meta": "1.00"}, {"id": "conceptual_art", "text": "conceptual_art", "meta": "0.00"}, {"id": "dark_fantasy", "text": "dark_fantasy", "meta": "0.00"}, {"id": "fantasy_realism", "text": "fantasy_realism", "meta": "1.00"}, {"id": "fashion", "text": "fashion", "meta": "0.00"}, {"id": "graffiti", "text": "graffiti", "meta": "0.00"}, {"id": "illustration", "text": "illustration", "meta": "0.00"}, {"id": "interior_design", "text": "interior_design", "meta": "0.00"}, {"id": "logo", "text": "logo", "meta": "0.00"}, {"id": "painting", "text": "painting", "meta": "1.00"}, {"id": "photography", "text": "photography", "meta": "0.00"}, {"id": "portrait_photography", "text": "portrait_photography", "meta": "0.00"}, {"id": "poster", "text": "poster", "meta": "0.00"}, {"id": "product", "text": "product", "meta": "0.00"}, {"id": "sticker", "text": "sticker", "meta": "0.00"}, {"id": "surrealism", "text": "surrealism", "meta": "0.00"}, {"id": "typography", "text": "typography", "meta": "0.00"}, {"id": "ukiyo-e", "text": "ukiyo-e", "meta": "0.00"}, {"id": "vector_design", "text": "vector_design", "meta": "0.00"}, {"id": "watercolor", "text": "watercolor", "meta": "0.00"}, {"id": "wildlife_photography", "text": "wildlife_photography", "meta": "0.00"}], "accept": ["3d_render", "cinematic", "fantasy_realism", "painting"], "tokens": [{"text": "matte", "start": 0, "end": 5, "id": 0, "ws": true}, {"text": "painting", "start": 6, "end": 14, "id": 1, "ws": false}, {"text": ",", "start": 14, "end": 15, "id": 2, "ws": false}, {"text": "fantasy", "start": 15, "end": 22, "id": 3, "ws": true}, {"text": "forest", "start": 23, "end": 29, "id": 4, "ws": true}, {"text": "of", "start": 30, "end": 32, "id": 5, "ws": true}, {"text": "glowing", "start": 33, "end": 40, "id": 6, "ws": true}, {"text": "trees", "start": 41, "end": 46, "id": 7, "ws": true}, {"text": "and", "start": 47, "end": 50, "id": 8, "ws": true}, {"text": "glowing", "start": 51, "end": 58, "id": 9, "ws": true}, {"text": "flying", "start": 59, "end": 65, "id": 10, "ws": true}, {"text": "insects", "start": 66, "end": 73, "id": 11, "ws": false}, {"text": ",", "start": 73, "end": 74, "id": 12, "ws": true}, {"text": "a", "start": 75, "end": 76, "id": 13, "ws": true}, {"text": "lake", "start": 77, "end": 81, "id": 14, "ws": true}, {"text": "reflecting", "start": 82, "end": 92, "id": 15, "ws": true}, {"text": "the", "start": 93, "end": 96, "id": 16, "ws": true}, {"text": "lights", "start": 97, "end": 103, "id": 17, "ws": false}, {"text": ",", "start": 103, "end": 104, "id": 18, "ws": true}, {"text": "myst", "start": 105, "end": 109, "id": 19, "ws": true}, {"text": "on", "start": 110, "end": 112, "id": 20, "ws": true}, {"text": "the", "start": 113, "end": 116, "id": 21, "ws": true}, {"text": "air", "start": 117, "end": 120, "id": 22, "ws": false}, {"text": ",", "start": 120, "end": 121, "id": 23, "ws": true}, {"text": "cinematic", "start": 122, "end": 131, "id": 24, "ws": true}, {"text": "feel", "start": 132, "end": 136, "id": 25, "ws": false}, {"text": ",", "start": 136, "end": 137, "id": 26, "ws": true}, {"text": "hyper", "start": 138, "end": 143, "id": 27, "ws": true}, {"text": "realistic", "start": 144, "end": 153, "id": 28, "ws": false}, {"text": ",", "start": 153, "end": 154, "id": 29, "ws": true}, {"text": "high", "start": 155, "end": 159, "id": 30, "ws": true}, {"text": "detail", "start": 160, "end": 166, "id": 31, "ws": false}, {"text": ",", "start": 166, "end": 167, "id": 32, "ws": true}, {"text": "blender", "start": 168, "end": 175, "id": 33, "ws": true}, {"text": "render", "start": 176, "end": 182, "id": 34, "ws": true}, {"text": ".", "start": 183, "end": 184, "id": 35, "ws": false}], "_annotator_id": "gpt35_turbo", "_session_id": "gpt35_turbo", "answer": "accept"}
{"text": "matte painting,fantasy forest of glowing trees and glowing flying insects, a lake reflecting the lights, myst on the air, cinematic feel, hyper realistic, high detail, blender render .", "_input_hash": -2110223808, "_task_hash": 813863613, "options": [{"id": "3d_render", "text": "3d_render", "meta": "0.00"}, {"id": "portrait_photography", "text": "portrait_photography", "meta": "0.01"}, {"id": "dark_fantasy", "text": "dark_fantasy", "meta": "0.00"}, {"id": "painting", "text": "painting", "meta": "0.01"}, {"id": "typography", "text": "typography", "meta": "0.00"}, {"id": "illustration", "text": "illustration", "meta": "0.02"}, {"id": "fantasy_realism", "text": "fantasy_realism", "meta": "0.96"}, {"id": "surrealism", "text": "surrealism", "meta": "0.01"}, {"id": "logo", "text": "logo", "meta": "0.00"}, {"id": "poster", "text": "poster", "meta": "0.00"}, {"id": "fashion", "text": "fashion", "meta": "0.01"}, {"id": "sticker", "text": "sticker", "meta": "0.00"}, {"id": "product", "text": "product", "meta": "0.00"}, {"id": "cinematic", "text": "cinematic", "meta": "0.03"}, {"id": "watercolor", "text": "watercolor", "meta": "0.00"}, {"id": "character_design", "text": "character_design", "meta": "0.00"}, {"id": "vector_design", "text": "vector_design", "meta": "0.00"}, {"id": "wildlife_photography", "text": "wildlife_photography", "meta": "0.00"}, {"id": "architecture", "text": "architecture", "meta": "0.00"}, {"id": "interior_design", "text": "interior_design", "meta": "0.00"}, {"id": "abstract_art", "text": "abstract_art", "meta": "0.00"}, {"id": "graffiti", "text": "graffiti", "meta": "0.02"}, {"id": "photography", "text": "photography", "meta": "0.00"}, {"id": "conceptual_art", "text": "conceptual_art", "meta": "0.02"}, {"id": "anime", "text": "anime", "meta": "0.00"}, {"id": "ukiyo-e", "text": "ukiyo-e", "meta": "0.00"}], "accept": ["fantasy_realism"], "tokens": [{"text": "matte", "start": 0, "end": 5, "id": 0, "ws": true}, {"text": "painting", "start": 6, "end": 14, "id": 1, "ws": false}, {"text": ",", "start": 14, "end": 15, "id": 2, "ws": false}, {"text": "fantasy", "start": 15, "end": 22, "id": 3, "ws": true}, {"text": "forest", "start": 23, "end": 29, "id": 4, "ws": true}, {"text": "of", "start": 30, "end": 32, "id": 5, "ws": true}, {"text": "glowing", "start": 33, "end": 40, "id": 6, "ws": true}, {"text": "trees", "start": 41, "end": 46, "id": 7, "ws": true}, {"text": "and", "start": 47, "end": 50, "id": 8, "ws": true}, {"text": "glowing", "start": 51, "end": 58, "id": 9, "ws": true}, {"text": "flying", "start": 59, "end": 65, "id": 10, "ws": true}, {"text": "insects", "start": 66, "end": 73, "id": 11, "ws": false}, {"text": ",", "start": 73, "end": 74, "id": 12, "ws": true}, {"text": "a", "start": 75, "end": 76, "id": 13, "ws": true}, {"text": "lake", "start": 77, "end": 81, "id": 14, "ws": true}, {"text": "reflecting", "start": 82, "end": 92, "id": 15, "ws": true}, {"text": "the", "start": 93, "end": 96, "id": 16, "ws": true}, {"text": "lights", "start": 97, "end": 103, "id": 17, "ws": false}, {"text": ",", "start": 103, "end": 104, "id": 18, "ws": true}, {"text": "myst", "start": 105, "end": 109, "id": 19, "ws": true}, {"text": "on", "start": 110, "end": 112, "id": 20, "ws": true}, {"text": "the", "start": 113, "end": 116, "id": 21, "ws": true}, {"text": "air", "start": 117, "end": 120, "id": 22, "ws": false}, {"text": ",", "start": 120, "end": 121, "id": 23, "ws": true}, {"text": "cinematic", "start": 122, "end": 131, "id": 24, "ws": true}, {"text": "feel", "start": 132, "end": 136, "id": 25, "ws": false}, {"text": ",", "start": 136, "end": 137, "id": 26, "ws": true}, {"text": "hyper", "start": 138, "end": 143, "id": 27, "ws": true}, {"text": "realistic", "start": 144, "end": 153, "id": 28, "ws": false}, {"text": ",", "start": 153, "end": 154, "id": 29, "ws": true}, {"text": "high", "start": 155, "end": 159, "id": 30, "ws": true}, {"text": "detail", "start": 160, "end": 166, "id": 31, "ws": false}, {"text": ",", "start": 166, "end": 167, "id": 32, "ws": true}, {"text": "blender", "start": 168, "end": 175, "id": 33, "ws": true}, {"text": "render", "start": 176, "end": 182, "id": 34, "ws": true}, {"text": ".", "start": 183, "end": 184, "id": 35, "ws": false}], "_annotator_id": "spacy_model", "_session_id": "spacy_model", "answer": "accept"}
The problem occurs in the review phase. Even though the input_hash and task_hash's line up, the annotations aren't being shown in the same review step. It shows the annotation for the same text for each annotator one right after the other.
Here's the command I'm running for review:
!PRODIGY_LOGGING=verbose PRODIGY_CONFIG_OVERRIDES="{\"choice_style\": \"multiple\"}" prodigy review gpt35-test100_final gpt35-test100_in3 --view-id choice
Am I missing something? The review functionality seems broken. I'm expecting to see an interface more like what was shown in the tutorial for NER where both annotations would be shown side by side.
