Duplicate annotations in output

[ines (Ines Montani)]

@leetdavid Thanks for the report, I've merged the post onto a previous thread since it seems to be related to the same problem/questions.

One thing to look out for is whether the work stealing timeout might be the culprit and whether your annotation workflow hits scenarios where previously unanswered questions are added back to the queue too early:

@ines

Another thing to keep an eye out for in your multi-annotator setup is whether you end up in scenario where annotators stop working before submitting their annotations and triggering the "work stealing" mechanism. If an annotator requests a batch but doesn't annotate it for a longer period of time (e.g. if they just close the app) it will eventually be sent out again to the next available annotator, to make sure you don't end up losing a batch if an annotator doesn't annotate it.

@ines

If you enable PRODIGY_LOGGING=basic, you should see a log message FEED: re-adding open tasks to stream whenever tasks are re-added for the next annotators. This should give you an idea whether this happens often in your scenario.