running prodigy on internal network with multiple annotators




Quite a naive question I am afraid - I want to run Prodigy off an internal server that many annotators could use and have it set up so annotators must log in/authenticate and that they would all be storing to their own datasets as opposed to a single, common dataset. Is this possible, and is it a particularly challenging approach? (I am a data scientist but I do have a software team who can help me if needed.)

My issue is that the text to be annotated is highly technical and the average Joe will struggle to accurately annotate the documents. However, at the moment I’ve got it running on a linux VM and the SME’s are not going to be comfortable running off the command line either - so I need to find a solution where they do not have to touch anything but a web browser, and I can still control the datasets and who is annotating what. Fingers crossed you have the solution.


(Ines Montani) #2

Not a native question at all! How you scale up your annotation projects and manage your annotators always depends on the use case, existing infrastructure and internal requirements. A powerful, general-purpose solution for this is something we’re actively working on for the Annotation Manager, which will be an extension product to Prodigy. However, we always wanted to make sure that workflows like this are also possible to implement with the regular Prodigy developer tool – you just have to do the setup and scaffolding yourself.

The easiest solution would be to start multiple Prodigy sessions on your VM, each with their own dataset (e.g. annotator A gets dataset_a, and so on) on a different host/port. If you want the annotators to be able to start their own sessions, you could run a service in front of Prodigy that takes care of that and maybe exposes a login-style form. The annotator could then access an internal URL via their browser, and enter their details. Your service then authenticates them, starts up a new Prodigy session with a custom dataset name on a free port, and returns the URL (or redirects the user accordingly).

Here’s some pseudocode for a possible endpoint:

def start_session(user_id, password):
    if is_authenticated(user_id, password):
        print('New session', user_id,
        dataset_name = get_dataset_name(user_id)
        port = get_free_port()
        start_prodigy_server(dataset_name, port=port)
        msg = 'Started Prodigy server at: http://your-host:{}'.format(port)
        return {'status': 200, 'msg': msg}
    return {'status': 530, 'msg': 'Authenticatio failed' }

If you search the forum for “multiple annotators”, you can also find other solutions that users have implemented. You might also want to check out this extension library by fellow Prodigy user @andy, which adds multi-user support and some other cool features like analytics:

(Tyler Watson) #3

What about for a situation where you have multiple annotators making annotations simultaneously on the same dataset for one or more NER tags? I don’t need to split up the datasets since I’m not concerned about annotation accuracy, and with one NER tag it wouldn’t make sense to create a new service for each tag.

(Ines Montani) #4

@tylernwatson That should just work! If you start a Prodigy instance and then have multiple users access the same link, it’ll just keep sending out batches of questions to whoever connects to it until all data is sent out.

(One thing to keep in mind: The workflow probably works well if you’re doing manual annotation. If you’re annotating with a model in the loop and have multiple people connecting to it, every annotator could potentially move the model in a different direction and causing it to make worse suggestions as a result. )

(Tyler Watson) #5

Thanks @ines! Concerning your second paragraph - I am hoping to have annotators use the ner.teach recipe to accept/reject NER spans the model is suggesting for one named entity based on a patterns jsonl file. When you say annotators could potentially move the model in a different direction, what do you mean? Is it that annotators might be applying two different understandings of what is(n’t) a given named entity and accepting/rejecting according to inconsistent rules? Assuming an ideal world where all annotators have a clear understanding of what qualifies as a given entity, would this still be the case?

(Ines Montani) #6

@tylernwatson Yeah, pretty much. In a single-user workflow with a model in the loop, the model progressively receives batches of updates, adjusts its weights, sends out new suggestions and so on. If you have five people annotating at the same time, the model might receive three batches in a very short period of time. And if you have one person who makes different decisions than the others, you can end up with a model in a weird state. And that model will then subsequently send out lower quality suggestions to everyone and their answers will then be less relevant, too. All of this is very subtle and in the end, you’ll have one dataset and no way of knowing what exactly happened and why the model isn’t converging.

But yes, in an ideal world where everyone makes exactly the same decisions, this wouldn’t really be a problem. If you care less about the model and more about the patterns, you might also want to give ner.match a try. This will show your annotators the pattern matches and ask for binary feedback, and will go through the input data as it comes in, with no model in the loop and no filtering.