Tasks are duplicated

[ryanwesslen]

hi @auguststapput!

Thanks for your question and welcome to the Prodigy community

@auguststapput

Relevant to this, config.jsonl is set to the following:

"feed_overlap": false
"exclude_by": "input"

I have now received the following log and one of my annotators has informed me that they have been given the same task that they have done earlier (using the same /?session=user_name):

Could it be work-stealing? Did you notice these duplicates near the end of the stream?

We fixed a front-end bug was fixed in v1.11.9, which given you're using v1.11.11. But work stealing is still a possibility as we outlined in our v1.11.9 announcement:

It should be be noted though, that a small number of duplicates is still expected in multi-user workflows with feed overlap set to false. This is perfectly normal behavior and should only occur towards the end of the example stream.

These "end-of-queue" duplicates come from the work-stealing mechanism in the internal Prodigy feed. "Work-stealing" is a preventive mechanism to avoid records in the stream from being lost when an annotator requests a batch of examples to annotate, effectively locking those examples, and then never annotates them. This mechanism allows annotators that reach the end of a shared stream to annotate these otherwise locked examples that other annotators are holding on to. Essentially we have prioritized annotating all examples in your data stream at least once vs at most once while potentially losing a few.

One way to check this is if you see FEED: re-adding open tasks to stream in the logs:

Repeated Examples in Multi-user Sessions

We can expose the settings for the timeout and that's probably a good idea – although, you typically want to at least have some level of work stealing enabled because otherwise, an annotator opening the app, looking around and then closing it again will lead to a batch not being annotated, which is typically not what you want. If you enable PRODIGY_LOGGING=basic, you should see a log message FEED: re-adding open tasks to stream whenever tasks are re-added for the next annotators. This should g…

One way to minimize the effect of work stealing is to inform your annotators to double-check that they save their most recent batch when they're done annotating.

In our Prodigy v1.12 alpha, one of the enhancements is more control/customization for task routing like the ability to turn off work_stealing via configuration allow_work_stealing:

Prodigy 1.12 alpha release: LLM-assisted workflows, prompt engineering & fully custom task routing for multi-annotator scenarios.

Hey everyone! We are really excited to share that we have just released the alpha version of Prodigy v1.12! (v1.12a1). This release is available for download for all v.1.11.x license holders and includes: New recipes for LLM-assisted annotations and prompt engineering: the LLM assisted workflows we have announced a while ago are now fully integrated with Prodigy and available out of the box. For v1.12a1 you'd still be restricted to OpenAI API to use them, but by v1.12 we definitely want to l…

Extended, fully customizable support for multi-annotator workflows . You can now customize what should happen when a new annotator joins an ongoing annotation project, how tasks should be allocated between existing annotators, and what should happen when one annotator finishes their assigned tasks before others. For common use-cases, you can use the options feed_overlap , annotations_per_task and allow_work_stealing (see the updated configuration documentation for details). Custom recipes can specify session_factory and task_router callbacks for full control.

But recall, that if you turn this off, you may now open yourself up to risks of losing some tasks getting lost in your stream.

It's worth noting that for our future v2 release, we're working to refactor our feeds which would eliminate this trade-off.

Repeated Examples in Multi-user Sessions

Hi @adin786, Eventually we decided not to expose this setting (it's set to 3600s) as it might potentially lead to unnecessary lags in the annotation flow to avoid what, in principle, is a good tradeoff. Instead, we've been refactoring the the feed logic to make sure there are fewer duplicates due to "work-stealing" by only triggering this mechanism once a session gets to the end of its queue. Effectively, there will be very few duplicates and only towards the end of the example stream. This ch…

For the v2 release, we are working on a complete redesign of feed mechanism that will eliminate the need for work-stealing tradeoff altogether.

In the meantime, if you want more control on task routing, I'd strongly recommend giving our v1.12 alpha a try (we'd appreciate the feedback!). Here's a preview of the task routing docs (link may change in the future).

Hope this helps!