Avoid tasks to be stolen between sessions

[TatyanaKavalenkaTR (Tatyana Kavalenka)]

Hello, I'm trying new custom task routing feature.
I faced with the issue that when all tasks are completed by annotator. I see following message : session1 has stolen item with hash -1652070420 from session2.
I need the task to be routed to session exactly I specified in task router. And do not want tasks to be stolen between sessions. How to avoid tasks to be stolen from other sessions ?

I use version: 1.12a4

[ryanwesslen]

hi @TatyanaKavalenkaTR,

Thanks for your question and welcome to the Prodigy community

@TatyanaKavalenkaTR

I faced with the issue that when all tasks are completed by annotator. I see following message : session1 has stolen item with hash -1652070420 from session2.
I need the task to be routed to session exactly I specified in task router. And do not want tasks to be stolen between sessions. How to avoid tasks to be stolen from other sessions ?

Do you mean work stealing?

We've also added to v1.12 the ability to turn off work stealing in the config. You can do this by setting allow_work_stealing: false. Here's a bit of background about work stealing:

Repeated Examples in Multi-user Sessions

Hi! Are you using the latest version of Prodigy v1.11.7? We recently shipped a fix for a problem that could cause duplicate batches to be sent out to different annotators under certain configurations. Another thing to keep an eye out for in your multi-annotator setup is whether you end up in scenario where annotators stop working before submitting their annotations and triggering the "work stealing" mechanism. If an annotator requests a batch but doesn't annotate it for a longer period of time …

Update: we've also a new section in our docs on work stealing:

In addition to upcoming material on v1.12, we're planning upcoming material that will deep dive more into topics like handling edge cases that can cause duplicates/missing data (and how to avoid them). We'd be grateful for any feedback or suggestions!

Hope this helps!

[TatyanaKavalenkaTR (Tatyana Kavalenka)]

@ryanwesslen Thank you very much! This option 'allow_work_stealing' did what I need!
I also noticed if user reach the end of stream, answers are not auto-saved. So user see the message "No tasks available" and had to press Save icon manually. So it will be easy for user to forget to press Save . Is there an option to Save answers when end stream reached ?

[ryanwesslen]

@TatyanaKavalenkaTR

Is there an option to Save answers when end stream reached ?

This has come up before:

Auto save when no more tasks left

I just loaded an examples.jsonl with a single example and was able to confirm that when you're presented with this interface after a single annotation. [CleanShot 2022-10-26 at 11.04.36] When I shut down Prodigy after this, indeed no examples are saved. Quick Fix I'd need to check with the Prodigy team about this though. Part of me feels like it could be a feature to automatically save the examples when you hit the end of the queue, but it also feels a bit like an anti-pattern because the i…

As he mentions, we don't have a ready-to-go option but do have an internal ticket to consider this. Thanks for the feedback as we'll add your request to the ticket. I'll post back if we make a fix. But unfortunately for now, I think just reminding annotators to click the save button is the best option.