Does "annotations_per_task" define annotations or annotators?

Hi there,

unfortunately I was unable to find an appropriate answer to the above question.

I have a task for three annotators, each sample should have a total of 2 annotations from 2 DIFFERENT annotators.

I defined '"annotations_per_task": 2' in prodigy.json and also defined the environment variable PRODIGY_ALLOWED_SESSIONS="andrew,berta,curt".

It is still unclear to me if, should an annotator have annotated each example once, they would be presented with examples they already annotated. In other words: do I need a custom router to ensure that the maximum annotators per example is equal to 1?

Clarifications very much appreciated.

Hi @nicolaiberk,

If you set "annotations_per_task": 2 and define the sessions up front like you did, each task should receive 2 annotations from different annotators. The built-in router makes a check against the DB and the current queues of the sessions to make sure the annotators do not see tasks they have already annotated.

If, on the other hand, you have 3 annotators but want each task to be annotated only once to maximize the parallelization of your annotation project, you can set "feed_overlap": false which technically is equivalent to setting "annotations_per_task": 1

(In case you haven't seen it yet, here's our explainer video for task routers that gets into the logic behind each setting: https://youtu.be/vyOtq-UXP-E)

1 Like

Perfect, thank you for the clarification!