Hi!
I just read that you plan to release Prodigy Annotation Manager for managing tasks for multiply annotators – is there any projected due date for this feature?
Currently we looking for a tool for managing multiply machine learning tasks between tens of annotators and we have two options – try to adapt your tool or investigate into option of writing our own tool in next two months, so projected date will help us to choose best option
We’d probably be looking at mid next year for the annotation manager, so you should probably build your own tooling if you need something sooner than that. I hope you’ll find Prodigy a good basis for your work. There are several ways you could work with multiple annotators, depending on your requirements.
In the basic case where you run Prodigy on your machine and do some annotations by yourself, you’ve got two queues, each with one producer and one consumer. One queue gives the questions, and the other receives the answers. If you have multiple annotators, then the forward queue is single producer, multiple consumer, and the backward queue is multiple producer, single consumer. So: how to structure that?
It’s often useful to let the server hold some state that’s specific to the annotation session. To do this, you can let each annotator talk to a single Prodigy service, and have those Prodigy services communicating with an upstream endpoint. This lets the Prodigy service stay single consumer, single producer, so you don’t have to make any changes within Prodigy itself. You just need to stream tasks from a local endpoint you provide. To get the annotations as they’re produced, the upstream service would be pulling changes from the annotation instances’ data stores.
The upstream service is the part you would have to implement yourself, based on your own requirements. It needs to send the work to the annotator instances. You may or may not need to fetch the annotations in real-time — it depends on whether you need the annotation queue to react to the decisions. If you don’t need that (e.g. if you can reconcile the annotations in a background process), the upstream service can be very simple.
Hope that helps!
If you could provide some codeblock to show the flow ,it would be really helpful.Thank You!
@akshitasood63 There are a few threads on the forum discussing custom workflows with multiple annotators – see this search query for more examples. You might also find this post helpful, which describes a custom use case of Prodigy at scale with multiple annotators:
On a related note, we've started work on the annotation manager and it's coming together very nicely so far 
I'm hoping we can share some work in progress soon!
Thanks a lot.I will be looking forward 