Bug with review recipe in 1.10.2+

,
1

In version 1.9.9 it used to be the case that when you have an example (same _task_hash/_input_hash) but 2 annotators disagreed on the label, and you used the review recipe to resolve the disagreement, it would show the disagreement in the following way:
Screenshot 2020-08-07 at 17.52.02

What seems to happen in version 1.10.2 and 1.10.3 is that it will first show you what the first annotator said...

Screenshot 2020-08-07 at 17.53.45
Screenshot 2020-08-07 at 17.53.451366×304 13.4 KB

...and after you reject/accept what the second annotator said:

Screenshot 2020-08-07 at 17.53.50
Screenshot 2020-08-07 at 17.53.501368×314 13.7 KB

This makes the review recipe less useful. Is this a bug or am I missing anything/doing anything wrong?
I execute the same command in both cases: prodigy review <output_table> <input_table>,<input_table>

2

Hi! That's very strange – I can't think of any relevant changes to the review recipe that could lead to this :thinking: It's definitely not intended – if both tasks have the same task hashes and different session IDs, they should be merged into a single question.

Are the screenshots you posted both using the same underlying input data but different Prodigy versions? Also, as a quick sanity check, could you try with with Prodigy v1.10.3?

3

the first screenshot I posted was taken a few weeks ago admittedly. But I just tried out version 1.9.9 again and everything still works fine. then installed version 1.10.3 and it doesn't. I am operating on the same database, so I just executed the pip install ... and then prodigy review .... Seems to be really a problem with 1.10.2/3

4

Thanks for checking – and I was mostly interested in whether it's the exact same data being rendered here or different examples (that could have ended up with different hashes and different sessions, depending on how they were annotated).

Could you share the JSON of those examples created by the review recipe?

5

Not quite sure how to get the json, but let me show you what is stored in the database. I first launched the 2 annotation tasks and saved the results in the tables sentence_boundary and sentence_boundary2.

I then ran the review recipe pointing to those 2 tables, saving the outputs in sentence_boundary_master.

Here is what is stored in the table sentence_boundary (using the prodigy db python api):

[{'html': 'Scientific methods<span style="background-color: #FFFF00">.</span> Genentech\'s steady focus on research yields a plump bottom line.',
  'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
  'boundary_index': 18,
  'boundary_char': '.',
  'source_id': 'PM17380983_0',
  '_input_hash': 2086794253,
  '_task_hash': -1768287807,
  '_session_id': None,
  '_view_id': 'html',
  'answer': 'accept'},
 {'html': 'Scientific methods. Genentech\'s steady focus on research yields a plump bottom line<span style="background-color: #FFFF00">.</span>',
  'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
  'boundary_index': 83,
  'boundary_char': '.',
  'source_id': 'PM17380983_1',
  '_input_hash': -1930479033,
  '_task_hash': -1197153440,
  '_session_id': None,
  '_view_id': 'html',
  'answer': 'accept'}]

Here is what is stored in sentence_boundary2:

  'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
  'boundary_index': 18,
  'boundary_char': '.',
  'source_id': 'PM17380983_0',
  '_input_hash': 2086794253,
  '_task_hash': -1768287807,
  '_session_id': None,
  '_view_id': 'html',
  'answer': 'reject'},
 {'html': 'Scientific methods. Genentech\'s steady focus on research yields a plump bottom line<span style="background-color: #FFFF00">.</span>',
  'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
  'boundary_index': 83,
  'boundary_char': '.',
  'source_id': 'PM17380983_1',
  '_input_hash': -1930479033,
  '_task_hash': -1197153440,
  '_session_id': None,
  '_view_id': 'html',
  'answer': 'reject'}]

And here is what is stored in sentence_boundary_master:

  'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
  'boundary_index': 18,
  'boundary_char': '.',
  'source_id': 'PM17380983_0',
  '_input_hash': 2086794253,
  '_task_hash': -1997274244,
  '_session_id': None,
  '_view_id': 'review',
  'answer': 'accept',
  'sessions': ['sentence_boundary'],
  'versions': [{'html': 'Scientific methods<span style="background-color: #FFFF00">.</span> Genentech\'s steady focus on research yields a plump bottom line.',
    'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
    'boundary_index': 18,
    'boundary_char': '.',
    'source_id': 'PM17380983_0',
    '_input_hash': 2086794253,
    '_task_hash': -1997274244,
    '_session_id': 'sentence_boundary',
    '_view_id': 'html',
    'answer': 'accept',
    'sessions': ['sentence_boundary'],
    'default': True}],
  'view_id': 'html'},
 {'html': 'Scientific methods<span style="background-color: #FFFF00">.</span> Genentech\'s steady focus on research yields a plump bottom line.',
  'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
  'boundary_index': 18,
  'boundary_char': '.',
  'source_id': 'PM17380983_0',
  '_input_hash': 2086794253,
  '_task_hash': 1330951531,
  '_session_id': None,
  '_view_id': 'review',
  'answer': 'accept',
  'sessions': ['sentence_boundary2'],
  'versions': [{'html': 'Scientific methods<span style="background-color: #FFFF00">.</span> Genentech\'s steady focus on research yields a plump bottom line.',
    'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
    'boundary_index': 18,
    'boundary_char': '.',
    'source_id': 'PM17380983_0',
    '_input_hash': 2086794253,
    '_task_hash': 1330951531,
    '_session_id': 'sentence_boundary2',
    '_view_id': 'html',
    'answer': 'reject',
    'sessions': ['sentence_boundary2'],
    'default': True}],
  'view_id': 'html'},
 {'html': 'Scientific methods. Genentech\'s steady focus on research yields a plump bottom line<span style="background-color: #FFFF00">.</span>',
  'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
  'boundary_index': 83,
  'boundary_char': '.',
  'source_id': 'PM17380983_1',
  '_input_hash': -1930479033,
  '_task_hash': -1793334956,
  '_session_id': None,
  '_view_id': 'review',
  'answer': 'accept',
  'sessions': ['sentence_boundary'],
  'versions': [{'html': 'Scientific methods. Genentech\'s steady focus on research yields a plump bottom line<span style="background-color: #FFFF00">.</span>',
    'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
    'boundary_index': 83,
    'boundary_char': '.',
    'source_id': 'PM17380983_1',
    '_input_hash': -1930479033,
    '_task_hash': -1793334956,
    '_session_id': 'sentence_boundary',
    '_view_id': 'html',
    'answer': 'accept',
    'sessions': ['sentence_boundary'],
    'default': True}],
  'view_id': 'html'},
 {'html': 'Scientific methods. Genentech\'s steady focus on research yields a plump bottom line<span style="background-color: #FFFF00">.</span>',
  'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
  'boundary_index': 83,
  'boundary_char': '.',
  'source_id': 'PM17380983_1',
  '_input_hash': -1930479033,
  '_task_hash': -555819029,
  '_session_id': None,
  '_view_id': 'review',
  'answer': 'accept',
  'sessions': ['sentence_boundary2'],
  'versions': [{'html': 'Scientific methods. Genentech\'s steady focus on research yields a plump bottom line<span style="background-color: #FFFF00">.</span>',
    'text': "Scientific methods. Genentech's steady focus on research yields a plump bottom line.",
    'boundary_index': 83,
    'boundary_char': '.',
    'source_id': 'PM17380983_1',
    '_input_hash': -1930479033,
    '_task_hash': -555819029,
    '_session_id': 'sentence_boundary2',
    '_view_id': 'html',
    'answer': 'reject',
    'sessions': ['sentence_boundary2'],
    'default': True}],
  'view_id': 'html'}]
6

Thanks, this is very helpful! :+1:

I still need to test this in more detail but at first glance, it looks like the hashes come back incorrectly when the review recipe re-hashes the examples to make sure the hashes are up to date :thinking: Initially, the hashes are indentical and reflect that both annotations are annotations on the same data, with the same suggestions. But in the final example, they're different, so they're treated as two different questions. My theory is that it might be including properties in the hashes that it shouldn't.

Here's a quick thing to try in the meantime: you can run prodigy stats to find the location of your Prodigy installation. Then open prodigy/recipes/review.py and change overwrite=True to overwrite=False (should be on line 113). If this solves the problem, it's the hashing config.

7

thanks, I tried what you suggested and it seems to have solved the problem!

8

Yay, glad it worked! I'm pretty sure it's related to the hashing of the keys in the JSON data here, so this should be easy to fix for the next version :slightly_smiling_face:

9

Update: Just released v1.10.4, which makes the hashing consistent when re-hashing examples in the review recipe. So you should be able to upgrade and see the same results you did before / without rehashing :slightly_smiling_face: