I am running the ner.correct recipe, but find that the same ~20 examples just cycle (and when they re-appear, they don't have the corrected labels).
I can confirm that this occurs with multiple models (both trained locally and not). My input jsonl does not have these repeats, and each line contains only a text field. The ~/.prodigy/prodigy.json is empty. I am on 1.11.3.
Thank you both for the reports. Looks like we pushed a bug along with a fix in version 1.11.3. This will be fixed in the next release (1.11.4) but in the meantime you can add a ?session={session_id} query parameter.
So if you're running Prodigy locally on port 8080 (which is default) then instead of doing your annotations at: http://localhost:8080, do your annotations at http://localhost:8080?session={session_id} (e.g. I use kab as my session id like: http://localhost:8080?session=kab.
We recommend using a session parameter whenever possible anyway but we will definitely get the fix out for allowing no session to be specified.
I think the issue is still there for v1.11.6. I'm using a custom recipe based on ner.manual (simply added one additional block). 10% of the examples are shown repeatedly.
@chuanw24 Does it make a difference if you save the results to a new dataset? This particular issue is incredibly mysterious because we've tried all kinds of things and have not been able to reproduce this, which makes it incredibly tricky