Task does not skip annotated examples

1

After updating to Prodigy 1.11.4 (from 1.11.3) the annotation tasks start from the beginning of the jsonl file instead of skipping the examples that have been annotated and saved. The dataset name and file are the same, in fact, in the Progress > Total count I can see that it counts the previous annotations.

2

Hi, that's definitely strange because very little changed between those two versions :thinking: Could you share a bit more about the recipe you're using and the type of data you're working with? Also, could you check whether the _input_hash and _task_hash values the recipe generates for the stream are the same as the hashes already in the dataset?

3

Hi. I'm using a custom recipe that has a choice and an html block. After annotating 2 examples, reopening and annotating 2 examples, Prodigy's output looks something like this:

{"text_id":"1","text":"test1","html": "a text to be displayed in html","options":[{"id":"OPTION1","text":"Option 1"},{"id":"OPTION2","text":"Option 2"},{"id":"OPTION3","text":"Option 3"}],"_input_hash":-277237864,"_task_hash":-2132303667,"_view_id":"blocks","config":{"choice_style":"single"},"accept":["OPTION1"],"answer":"accept","_timestamp":1632475796}

{"text_id":"2","text":"test2","html": "a text to be displayed in html","options":[{"id":"OPTION1","text":"Option 1"},{"id":"OPTION2","text":"Option 2"},{"id":"OPTION3","text":"Option 3"}],"_input_hash":-2007222786,"_task_hash":-1884570687,"_view_id":"blocks","config":{"choice_style":"single"},"accept":["OPTION2"],"answer":"accept","_timestamp":1632475797}

{"text_id":"1","text":"test1","html": "a text to be displayed in html","options":[{"id":"OPTION1","text":"Option 1"},{"id":"OPTION2","text":"Option 2"},{"id":"OPTION3","text":"Option 3"}],"_input_hash":-277237864,"_task_hash":-2132303667,"_view_id":"blocks","config":{"choice_style":"single"},"accept":["OPTION1"],"answer":"accept","_timestamp":1632475840}

{"text_id":"2","text":"test2","html": "a text to be displayed in html","options":[{"id":"OPTION1","text":"Option 1"},{"id":"OPTION2","text":"Option 2"},{"id":"OPTION3","text":"Option 3"}],"_input_hash":-2007222786,"_task_hash":-1884570687,"_view_id":"blocks","config":{"choice_style":"single"},"accept":["OPTION2"],"answer":"accept","_timestamp":1632475841}

As you can see I got the same 2 examples repeated, with the task and input hashes are the same and only the timestamp changes. It may be possible that it also happened with the version 1.11.3 and I didn't notice but I tried this same dataset & recipe with Prodigy 1.10.7 and this issue doesn't happen.

4

Hi Natalia, I'm investigating this issue and not able to reproduce it yet. Could you share a little more info?

Are you using the feed_overlap configuration?
Are you annotating with a ?session query parameter?

Thanks!