I have loaded 3 examples into prodigy to annotate. I have 3 users which in same time will annotate the 3 examples, user_1 annotate example_1, user_2 annotate example_2 and user_3 annotate example_3.
I set the "feed_overlap": false
, I used the multi user session mechanism by simply adding a session identifier to the annotation url e.g.,
(the example showed in the interface is example_2 )
Why user_1 and user_2 get the same example which is example_1 to annotate?, shouldn't be user_1 get example_1, user2 get example_2 and user_3 get example_3 ?
What you're observing might seem counterintuitive but it has its explanation.
Prodigy queues tasks in batches to each user session. I believe you use the default batch size in your trial which is 10. If the input stream is smaller than the batch size, which is the case here (just 3 inputs), Prodigy will try to fill in the queues as best as possible and that includes "stealing" work from other sessions. You can stop that behavior by setting allow_work_stealing to false in your prodigy.json.
If you do that (keeping batch size equal to 10), you'll see that all 3 questions will go to the first session and the other two sessions will get "no tasks available" message in the UI.
You can also set batch_size to 1 (or, alternative, set instant_submit to true), thus making it smaller than the source size to get the behavior you expect in this toy example.
Obviously, this toy example is an edge case and usually the input stream is much longer than the batch size. So in most cases you'd observe the right behavior.
Also, work_stealing is set to true by default as we belive it is a good thing to have in multi-annotator scenarios to prevent examples being locked up by idle sessions. It is probably better to have a few duplicates than lose examples altogether, but it can be completely disabled as mentioned above. You can read more about the motivation behind work stealing here.