the running locally is the limiting factor. Our prodigy instance is deployed on an EC2 instance and then exposed to our teams and vendors. We use named sessions to track work and aggregate labels/annotations by agent. We cannot have a local deployment for each user, they are not working in a technical environment.
We are considering doing something similar to our data files, and having multiple instances up and running to spread annotators across different instance URL. Then each dataset would be "swapped" to the other instance to collect adjudication labels.
A more detailed use case here for thinking about how to support it:
We have a dataset of 10K columnA and columnB pairs. We want a label on a 4 point scale for similarity. We want the document to appear as many times as it takes for 3 matching labels on 1 of the 4 point scales. Once a document receives 3 or more labels on 1 of the options, it is removed from the stream. This allows us to utilize an agent force of any size, without having to send every record to each agent.