# Repeated Examples in Multi-user Sessions **URL:** \ **Category:** Uncategorized\ **Tags:** streams\ **Created:** [January 25, 2022, 11:43pm UTC](https://support.prodi.gy/t/repeated-examples-in-multi-user-sessions/5235 "2022-01-25T23:43:09Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rz789](https://avatars.discourse-cdn.com/v4/letter/r/a9a28c/32.png) [@rz789](https://support.prodi.gy/u/rz789)\ **Post date:** [January 25, 2022, 11:43pm UTC](https://support.prodi.gy/t/repeated-examples-in-multi-user-sessions/5235/1 "2022-01-25T23:43:09Z")
Dear Ines and the Prodigy Team, My team and I have been using the Prodigy tool to help us with annotating multiple user text classification tasks. We set up a /?session=% username for each session and hosted a single instance of Prodigy. However, we found repeated annotating examples for the users. I have checked that our "\_input\_hash" and "\_task\_hash" for each example are different, as some of the forum discussion has suggested. I also set it up that the "feed overlap" and "force stream order" are "true" for our session. I would be happy to provide any additional examples/screenshots. Would really appreciate your help to give us some directions as to where we should check next! Thanks so much! ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [January 26, 2022, 3:50pm UTC](https://support.prodi.gy/t/repeated-examples-in-multi-user-sessions/5235/2 "2022-01-26T15:50:30Z")
Hi! Are you using the latest version of Prodigy v1.11.7? We recently shipped a fix for a problem that could cause duplicate batches to be sent out to different annotators under certain configurations. Another thing to keep an eye out for in your multi-annotator setup is whether you end up in scenario where annotators stop working before submitting their annotations and triggering the "work stealing" mechanism. If an annotator requests a batch but doesn't annotate it for a longer period of time (e.g. if they just close the app) it will eventually be sent out again to the next available annotator, to make sure you don't end up losing a batch if an annotator doesn't annotate it. ---
**Author:** ![rz789](https://avatars.discourse-cdn.com/v4/letter/r/a9a28c/32.png) [@rz789](https://support.prodi.gy/u/rz789)\ **Post date:** [January 27, 2022, 11:03pm UTC](https://support.prodi.gy/t/repeated-examples-in-multi-user-sessions/5235/3 "2022-01-27T23:03:05Z")
Dear Ines, Thanks so much again for your reply! We have upgraded to v1.11.7 now. In our case, we want to let the annotator to have some flexibility to the time it takes for them to annotate. So I wonder, can we turn off the ""work stealing" mechanism" as you mentioned above, and what effect would it cause for the multiple user sessions? Would you have other suggestions if we want the batch sent to a user to stay, rather than rolling to the next available annotator? Thanks so much again! ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [January 28, 2022, 10:21am UTC](https://support.prodi.gy/t/repeated-examples-in-multi-user-sessions/5235/4 "2022-01-28T10:21:25Z")
> [@rz789](#): > > So I wonder, can we turn off the ""work stealing" mechanism" as you mentioned above, and what effect would it cause for the multiple user sessions? Would you have other suggestions if we want the batch sent to a user to stay, rather than rolling to the next available annotator? We can expose the settings for the timeout and that's probably a good idea – although, you typically want to at least have some level of work stealing enabled because otherwise, an annotator opening the app, looking around and then closing it again will lead to a batch not being annotated, which is typically not what you want. If you enable `PRODIGY_LOGGING=basic`, you should see a log message `FEED: re-adding open tasks to stream` whenever tasks are re-added for the next annotators. This should give you an idea whether this happens often in your scenario. ---
**Author:** ![rz789](https://avatars.discourse-cdn.com/v4/letter/r/a9a28c/32.png) [@rz789](https://support.prodi.gy/u/rz789)\ **Post date:** [January 29, 2022, 7:36pm UTC](https://support.prodi.gy/t/repeated-examples-in-multi-user-sessions/5235/5 "2022-01-29T19:36:58Z")
Dear Ines, Thanks so much for your reply again! I understand. I have one more question, what is the current time that the stream will enable the stealing mechanism. I am currently planning to let the annotators know that for example, they have to be done with the number of cases by x amount of time, so that we manually avoid the roll over error. So that is why I have this question. Another thing I find is that we seems to be able to change the number of cases in a stream by the config file. Would you think any error would occur, if I set the number of cases of the batch to be say 4-5000s? Many thanks again! ---
**Author:** ![adin786](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/adin786/32/3284_2.png) [@adin786](https://support.prodi.gy/u/adin786)\ **Post date:** [January 20, 2023, 11:01am UTC](https://support.prodi.gy/t/repeated-examples-in-multi-user-sessions/5235/6 "2023-01-20T11:01:50Z")
Hi, can I follow up on this thread to ask whether this timeout setting was ever added as a config option. It seems like a good idea in case as you say, users may open the prodigy url, then close the tab, etc. Thanks ---
**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\ **Post date:** [January 20, 2023, 4:51pm UTC](https://support.prodi.gy/t/repeated-examples-in-multi-user-sessions/5235/7 "2023-01-20T16:51:36Z")
Hi @adin786, Eventually we decided not to expose this setting (it's set to 3600s) as it might potentially lead to unnecessary lags in the annotation flow to avoid what, in principle, is a good tradeoff. Instead, we've been refactoring the the feed logic to make sure there are fewer duplicates due to "work-stealing" by only triggering this mechanism once a session gets to the end of its queue. Effectively, there will be very few duplicates and only towards the end of the example stream. This change will be available in `v1.11.9` release scheduled next week. For the `v2` release, we are working on a complete redesign of feed mechanism that will eliminate the need for work-stealing tradeoff altogether. ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [January 24, 2023, 7:29pm UTC](https://support.prodi.gy/t/repeated-examples-in-multi-user-sessions/5235/8 "2023-01-24T19:29:45Z")
hi @adin786! As Madga mentioned, we just released v1.11.9 which refactors the logic. You should have received an email if your Prodigy license is still active. > [@Prodigy v1.11.9 release](https://support.prodi.gy/t/prodigy-v1-11-9-release/6263): > > We have just released Prodigy v1.11.9 that fixes the persistent problem of showing duplicate examples to annotators. This issue was affecting multi-annotator annotation workflows and high latency scenarios such as e.g. having Prodigy and the database deployed on a separate remote servers. This release eliminates the duplicates produced by a tricky front-end bug. It should be be noted though, that a small number of duplicates is still expected in multi-user workflows with [feed overlap](https://prodi.gy/docs/api-web-app#multi-user-sessions) set to fal… If you have questions, please respond back to that post as we're keeping an eye out for v1.11.9 related issues.