Hi, I am carrying out a project that involves two annotators. In my case, one annotator is expected to annotate more examples than the other since the time allocated for annotation is different for the two of them. I have run a small Prodigy test to see if both annotators would always find examples to annotate in their sessions while there are still examples without annotations remaining. However, I encountered an unexpected behaviour that I want to change. First I'll provide more info about my test run.
I ran Prodigy with the ner.manual recipe and created a new database with 150 examples of text. Before that I used export PRODIGY_ALLOWED_SESSIONS=joe,jane to create the sessions for the two annotators. The prodigy.json file only specified "annotations_per_task": 1.
I logged into session of annotator joe, and kept the A key pressed to quickly accept all examples (as if they had been annotated). Once it reached about 90 examples, Prodigy would tell annotator joe that there were no tasks available, even though there still remained 60 examples to annotate.
Next, I logged into the session of annotator jane. Let's assume that jane would be the annotator dedicating the least amount of time to annotating examples. I realized that if I logged into Jane's session and then log back into Joe's session, joe would have 10 new examples to annotate, which corresponds to the default batch_size. However, once joe annotates those 10 examples Prodigy would display the "No available tasks" message, even though there are more examples to be annotated. If I log back into Jane's session (or simply refresh the window with Jane's session), and the refresh the window with joe, Joe's session would get another set of 10 examples to annotate. In summary, if jane annotates sporadically, she will always have examples to annotate. But if joe is faster (because joe dedicates more time to annotating), joe will run out of examples quickly, and he will only get new ones once a browser window with jane's session is refreshed (even if that session doesn't annotate anything). Also, the new examples joe gets after this is only the number of batch_size, which is limiting and forces us to refresh a session of jane just so that joe can continue to annotate examples.
The behaviour I want to implement in Prodigy would be that joe and jane always have examples to annotate until every dataset entry has been annotated. I am aware this can lead to annotator imbalance, but it is ok in this case, since only one annotator will be able to annotate full time. How can I implement such behaviour?
Thank you very much for a detailed description of what you're observing. Let me first explain what is going on there.
There can be two reasons why joe is seeing "No tasks available" message before getting to the end of the input file:
Either the tasks have already been saved in the dataset used for this given run (e.g. from the previous test run)
Or janehas already consumed and saved one of her batches
If you start over with a fresh (target) dataset (no examples saved) and access the server only with joe URL you should be able to get to end of the queue i.e. 150th example.
Now, why there's this effect of examples appearing upon the refresh in the other session:
In Prodigy, examples are distributed to sessions (joeand jane) on pull i.e. each session "asks" for questions from the main queue. How much they can get in each pull depends on the batch size (10 by default).
To ensure that all examples get annotated, Prodigy, by default allows questions to be stolen by more eager sessions so that questions do not get locked up by idle sessions (cf. work_stealing).
Back to your example, if joe got to the end of his queue (for either of the 2 reasons mentioned above) but the last batch of his annotations has not been saved yet it means that it is in the buffer available for stealing. When janejoins she, effectively, steals these examples (you can observe that happening with logs set to PRODIGY_LOGGING=basic and looking for something like:
SESSION: test-jane has stolen item with hash 853456830 from test-joe
Now, if you refresh joesession, he will no longer have these questions stolen from him and if they are still not saved in the DB with annotation from jane he should be able to pull them back again, which is why there are questions available again.
Hope it make sense? I realize it might be confusing to see the "No examples available" screen and then have the examples available again upon refresh, but except for feed_overlap set to true it's impossible to know upfront if new questions would appear as it depends on the behavior of other sessions. The whole point of this back and forth is to make sure that all examples get at least some annotations. Another consequence of work_stealing is that it can lead to some occasional duplicates in the DB when both joe and janeend up annotating the same batch. It's by design, we considered occasional duplicates "less bad" than having examples with no annotation at all. This behavior can be turned off by setting allow_work_stealing to false in prodigy.json but from your description it looks like it's the behavior that you want.
To answer your question: your setting is correct. Instead of annotations_per_example you might also just set feed_overlap to false. The effect should be the same. If your annotators annotate for longer period you should get all examples annotated. It's true, though, that you should also instruct your annotators that whenever they get "No tasks available screen" they should save their answers (that makes them unstealable, at least with 1 annotation per example setting) and try refreshing just to check if there's anything "stealable" still. On our end, I think we should perhaps improve the "No tasks available screen" to remind of these two actions for multiuser workflows.
I ran another test, but this time instad of using "annotations_per_task": 1 in prodigy.json I used "feed_overlap" : false. The behaviour was different. When using feed_overlap I was able to run through all the 150 examples with just joe without having to refresh jane's session, as you mentioned in your answer. Shouldn't "annotations_per_task": 1 and "feed_overlap" : false behave the same for this case? I can't run through all 150 examples with joe when using "annotations_per_task": 1.
To clarify, each time I do a test run I start with a new database by dropping the previous one using prodigy drop ner_test.
The reason why I was using "annotations_per_task is because even though I'm setting it to 1 for now, I intend to set it to something like 1.5 in the future. The rationale is that I want to configure the following use case:
Let's say we have 150 examples. We would like to have the two annotators annotate all of them, but also have some overlap for assessing concordance between the annotators. Let's say I set "annotations_per_task": 1.2, which from what I understand would result in 180 annotations (150 x 1.2). Therefore, there would be 30 examples that would be annotated twice. Here are my questions:
Is there a way to ensure that the 30 examples that will have 2 annotations get such annotations from different annotators (in this case, jane and joe)?
Let's say joe will be a full-time annotator and jane a part time annotator. Is there a way to ensure that joe will always have a stream of examples? That could even mean that joe ends up annotating close to the full 150 examples, and jane only annotates the 30 examples that will have 2 annotations.
I have ran a test for jane and joe with a new database and 150 examples, setting "annotations_per_task": 1.2. Here's what happened:
When logging into joe's session and quickly accepting all the examples I'm only able to annotate up to 95 examples and then it shows the "No tasks available" message..
I then log in or refresh jane's session
Then refresh joe's session. joe gets examples again, but this time less than the batch_size of 10, he gets 6
joe accepts the 6 examples and saves, he gets "No tasks available again"
jane refreshes session (note jane is not annotating, just refreshing the session)
joe refreshes session. He gets only 3 examples now. He accepts and saves these examples
jane refreshes session again. joe refreshes session. joe gets only one example to annotate. He accepts the example and saves, gets "No tasks available"
jane refreshes session again. joe refreshes session. No more examples for joe, he gets "No tasks available"
At this point I can't get joe to annotate more examples. Total examples annotated are 105. I was expecting joe to be able to annotate up to 150 max, since the 30 examples that have to receive additional annotations due to "annotations_per_task": 1.2 should be reserved for jane to avoid duplicate annotations by the same annotator.
The observed difference between these two overlap settings is due to the way work_stealing is applied. In feed_overlap the work_stealing is applied until there are no unsaved examples left in any of the sessions. That is why you could have gotten to the end of the queue annotating with joe only. In the case of annotations_per_task the work stealing is applied only until the estimated target for joe has been reached which would be ca 75 examples (taking the 150 input). The remaining questions will be queued for jane until she gets to them.
The idea behind annotations_per_task setting is to approximate this target as closely as possible given the available pool of the annotators. If more than annotation per task is configured, it is expected there will be enough active annotators to reach that target. If you can't assume there will be enough active annotators to reach the target, it's better to use feed_overlap setting (especially if you have 2 annotators and want one annotator to annotated whatever the other annotator didn't).
About your second experiment with 1.2 annotations per task.
The 1.2 condition is applied on each pull to the main queue. Everytime joe asks for questions, the task router first tries to satisfy the whole number part of the fraction i.e. 1. The to handle the fraction part i.e. 0.2, it computes the probability where the task should be sent. Effectively, some tasks are sent to joe, some to alice and some to both.
If only joe is annotating, the probability of sending the task to jane would increase given the 1.2 condition and the fact that she is expected to annotate as well. This is why joe is allowed to steal less and less as he progresses (and this is why you're observing numbers smaller than the batch size).
It's a probability based mechanism because it's hard to know upfront how many you should send in total to one or another. Since the input files can be huge or have undefined size the Controller cannot know upfront the total, which is why it needs to be estimated on batch-basis. In other words, the task router takes "local" decisions trying to fulfill the conditions as best as possible given current conditions.
This post explains the mechanism a bit more: How does `annotations_per_task : 2.5` work.
Is there a way to ensure that the 30 examples that will have 2 annotations get such annotations from different annotators (in this case, jane and joe)?
The multiple annotations resulting from task router settins (overlap or annotations_per_example) always come from different annotators.
As mentioned above, especially the fraction value of annotations_per_task is the number of annotations per task that you can expect on average. Task router will try to fulfill this target as best as possible based on the progress of annotations. If all annotators defined in ALLOWED_SESSIONS are active, with enough number of annotations (cf. the proablistic based assignemnt) the final numbers will converge to the sttetting.
If you'd rather have a more precise router because you can afford precomputing the total or even the queues, you can always implement your custom task router, see here for some examples of how it can be done: Task Routing · Prodigy · An annotation tool for AI, Machine Learning & NLP
