ner.correct examples repeat

I am running the ner.correct recipe, but find that the same ~20 examples just cycle (and when they re-appear, they don't have the corrected labels).

I can confirm that this occurs with multiple models (both trained locally and not). My input jsonl does not have these repeats, and each line contains only a text field. The ~/.prodigy/prodigy.json is empty. I am on 1.11.3.

I'm also getting this same issue on 1.11.3 but I'm using ner.manual

Thank you both for the reports. Looks like we pushed a bug along with a fix in version 1.11.3. This will be fixed in the next release (1.11.4) but in the meantime you can add a ?session={session_id} query parameter.

So if you're running Prodigy locally on port 8080 (which is default) then instead of doing your annotations at: http://localhost:8080, do your annotations at http://localhost:8080?session={session_id} (e.g. I use kab as my session id like: http://localhost:8080?session=kab.

We recommend using a session parameter whenever possible anyway but we will definitely get the fix out for allowing no session to be specified.

1 Like

Just released v1.11.4, which should fix the underlying issue!

I think the issue is still there for v1.11.6. I'm using a custom recipe based on ner.manual (simply added one additional block). 10% of the examples are shown repeatedly.

@chuanw24 Does it make a difference if you save the results to a new dataset? This particular issue is incredibly mysterious because we've tried all kinds of things and have not been able to reproduce this, which makes it incredibly tricky :thinking: