# How to manage multiple annotators?

**URL:** <https://support.prodi.gy/t/how-to-manage-multiple-annotators/6643>\
**Category:** Uncategorized\
**Tags:** usage, textcat\
**Created:** [June 30, 2023, 2:25pm UTC](https://support.prodi.gy/t/how-to-manage-multiple-annotators/6643 "2023-06-30T14:25:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sudarshan85](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sudarshan85/32/2199_2.png) [@sudarshan85](https://support.prodi.gy/u/sudarshan85)\
**Post date:** [June 30, 2023, 2:25pm UTC](https://support.prodi.gy/t/how-to-manage-multiple-annotators/6643/1 "2023-06-30T14:25:58Z")

</div>

Hello,  
I am working on a project involving multiple annotators using Prodigy to create a gold standard dataset for validating an NLP model. Specifically, we have a rule-based model that analyzes clinical notes and extracts snippets from the note based on the presence of a particular concept of interest. We then have subject matter experts (SMEs) use Prodigy to annotate the same snippet using the `textcat.manual` recipe as binary classification task with the labels `TRUE` and `FALSE`.

The environment is highly restricted due to the nature of the data and has no access to the internet and the SMEs that I work with are not technical. Thus, I have made launching and using Prodigy as easy as possible. This is our current workflow:

1. I first extract the snippets output by the model into a format that is required by Prodigy, each concept of interest will get its own jsonl file
2. I have a bash script that installs all the dependencies and asks the user for the concept that they would like to annotate and launch Prodigy (through a Python script that runs the recipe)
3. This Prodigy session launches with a specific `prodigy.json` that is written out to disk by the Python script since it contains user specific information
4. Each concept and user get its own database and within it each session gets its own dataset starting with `ds_1` and incrementing for each session
5. Once a reasonable number of snippets has been annotated by two annotators for the same concept, I process them and calculate the interrator agreement and other metrics to refine the annotation guidelines.

I have slightly modified the default Prodigy interface. Here is a screenshot of the interface that I've shared in my instructions to the SMEs:

![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/4/402faa6e93d3b20f0fdab30f80b8e500fb4a98e5.png)

When I perform post process, I just specify the path to the database and its name in `db_settings` and extract all the annotated snippets. This seems to work well, but there might be better way to handle multiple annotators.

What we want is an easy way for annotators to discuss specific snippets. For example, if annotator 1 has a problem with a specific snippet, they can note the snippet ID and share it with annotator 2. Annotator 2 can then look up that snippet on Prodigy.

Questions:

1. Is there a better way to implement what I've described than the way I've implemented it? I have to work around heavy restrictions
2. Is there way to search snippets within Prodigy so that multiple annotators can collaborate across snippets to develop annotation guidelines before annotating "for real" to move forward.

Thank you.

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [June 30, 2023, 7:12pm UTC](https://support.prodi.gy/t/how-to-manage-multiple-annotators/6643/2 "2023-06-30T19:12:39Z")

</div>

hi @sudarshan85!

First off - great work! This is a fascinating workflow and especially impressive under a highly secure environment.

> [@sudarshan85](#):
>
> Is there a better way to implement what I've described than the way I've implemented it? I have to work around heavy restrictions

Several thoughts. Let's start with your core question:

> [@sudarshan85](#):
>
> What we want is an easy way for annotators to discuss specific snippets. For example, if annotator 1 has a problem with a specific snippet, they can note the snippet ID and share it with annotator 2. Annotator 2 can then look up that snippet on Prodigy.

## Step 1: Enabling flagging

Have you considered using flagging for annotators to "flag" problems with each record?

Just add this to your global `prodigy.json`: `"show_flag": true`

@koaning has a great tutorial on this:

[![](https://img.youtube.com/vi/D8KkXee2x5o/maxresdefault.jpg "Flagging while Annotating - Prodigy SHORTS") ](https://www.youtube.com/watch?v=D8KkXee2x5o)

With this, annotators can identify problems at the moment, then keep moving on.

## Step 2: (Optional) Add in input text for which annotator to send and message for them

You could also create a custom interface with `blocks`. If you created two input texts, one for who to send to (the other annotator) and the other for text to state the message/explanation.

Something like this:

![Export-1688150368396](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/3/3ba08fc81411f825c4f7352feb46c2e4ee8314e2.gif)

```python
import prodigy
from prodigy.components.preprocess import add_tokens
import requests
import spacy

@prodigy.recipe("cat-facts")
def cat_facts_ner(dataset, lang="en"):
    # We can use the blocks to override certain config and content, and set
    # "text": None for the choice interface so it doesn't also render the text
    blocks = [
        {"view_id": "classification", "label": "concept_of_interest"},
        {"view_id": "text_input", "field_id": "send_to", "field_label": "Send annotation to:", "field_suggestions": ["Steve", "Cindy", "Oliver", "Deepak"]},
        {"view_id": "text_input", "field_id": "comments", "field_rows": 3, "field_label": "Explain your decision"}
    ]

    def get_stream():
        res = requests.get("https://cat-fact.herokuapp.com/facts").json()
        for fact in res:
            yield {"text": fact["text"]}

    nlp = spacy.blank(lang) # blank spaCy pipeline for tokenization
    stream = get_stream() # set up the stream
    stream = add_tokens(nlp, stream) # tokenize the stream for ner_manual

    return {
        "dataset": dataset, # the dataset to save annotations to
        "view_id": "blocks", # set the view_id to "blocks"
        "stream": stream, # the stream of incoming examples
        "config": {
            "blocks": blocks # add the blocks to the config
        }
    }

```

Vincent has created another awesome video just on that:

[![](https://img.youtube.com/vi/lZdM2HScvVo/maxresdefault.jpg "Custom Interfaces via Blocks - Prodigy Shorts") ](https://www.youtube.com/watch?v=lZdM2HScvVo)

Perhaps you could try to use some custom javascript to only reveal the blocks (input\_text) when the item is flagged.

## Step 3: Use the flagged annotations to other annotators

Now each flagged example will be identified in the DB along with who to send it to and the text, you would need to write up a script to serve up those examples as a stream to that annotator. This will depend a bit on your workflow but the simplest way is to export out flagged examples to separate `.jsonl` files for each annotator the flagged examples are intended for secondary review.

You could then repeat this and if you wind up getting no resolution (e.g., the 2nd annotator agrees there's something wrong), create a new route to use the `review` recipe to review 2+ reviews. This could be some arbiter like a manager or the most senior teammate.

> [@sudarshan85](#):
>
> Is there way to search snippets within Prodigy so that multiple annotators can collaborate across snippets to develop annotation guidelines before annotating "for real" to move forward.

## Step 4: Update the Annotations Guidelines

One other suggestion is that perhaps examples that meet some quality (e.g., maybe the 2nd reviewer also flags that example), save as examples that you add to your annotation guidelines, which can be shown in Prodigy with `instructions: "path/to/my_page.html"`. Perhaps you could use a `.jinja` template to automatically populate the instructions html.

We wrote a case study where the Guardian had a similar follow up with the ultimate guide to help improve their annotation guidelines:

> **[How the Guardian approaches quote extraction with NLP · Explosion](https://explosion.ai/blog/guardian#workflow)**
>
> A case study of the Guardian's spaCy-Prodigy workflow to modularize quote extraction for content creation. This study includes iterative annotation guidelines and custom interface functionality.

Perhaps the post can help SMEs who are new to an NLP workflow could appreciate the workflow (e.g., what are annotation guidelines used for, an example of flagging, etc.).

> [@sudarshan85](#):
>
> This Prodigy session launches with a specific `prodigy.json` that is written out to disk by the Python script since it contains user specific information

Just curious - are you aware of the difference between your global and local `prodigy.json` and overrides?

> When you run Prodigy, it will first check if a **global configuration file** exists. It will also check the **current working directory** for a `prodigy.json` or `.prodigy.json` . This allows you to overwrite specific settings on a project-by-project basis.

Make sure to use all three for the three levels of your service.

- global: Settings for all users (e.g., all users are connected to the right database)
- user: Setting for each user; can also be thought of as "project" based as it applies to any tasks run by the user
- task: Setting on the task level for each user

> [@sudarshan85](#):
>
> Each concept and user get its own database and within it each session gets its own dataset starting with `ds_1` and incrementing for each session

Sure that's definitely an option. Just want to make sure - are you aware of [named-multi user sessions](https://prodi.gy/docs/api-web-app#multi-user-sessions)? This approach tends to be the more popular and default behavior for multiple annotators. But running a 1 dataset / 1 port per user is another option. Here's a good pro/con of each:

> [@Help - Multiple Annotators setup](https://support.prodi.gy/t/help-multiple-annotators-setup/3343/4):
>
> Ah, sorry if the terminology I used was confusing! What I meant was the session/instance/process. There are generally two ways you could set this up and it depends on your project, annotators and how you want to set up the workflow: Start one Prodigy process one a given host/port per annotator and save the annotations to separate datasets. For example, on port 8080, you start the Prodigy server with 100 documents and save to annotations\_andrew, on port 8081 you start Prodigy with the next 100 …

It's worth noting we're planning to make some of our biggest changes to Prodigy next week with v1.12 release. We currently have the candidate out now:

> [@Prodigy 1.12.0rc2 release candidate available for download!](https://support.prodi.gy/t/prodigy-1-12-0rc2-release-candidate-available-for-download/6636):
>
> In the last few weeks we've been collecting your feedback on the v1.12 alphas (thanks for giving it a spin!) and we've been ironing out all the outstanding issues. We are excited to announce that v1.12 release candidate (v1.12.0rc2) is now available for download! Like the alphas (and the upcoming v1.12 stable) this download is available to all v1.11x licence holders. As mentioned in our [previous post](https://support.prodi.gy/t/prodigy-1-12-alpha-release-llm-assisted-workflows-prompt-engineering-fully-custom-task-routing-for-multi-annotator-scenarios/6552), for v1.12 we have completely refactored Prodigy internals to make the annotation flow more tr…

First, I'd be curious of your thoughts on task routing:

> **[Task Routing · Prodigy · An annotation tool for AI, Machine Learning & NLP](https://future.prodi.gy/docs/task-routing/)**
>
> A downloadable annotation tool for LLMs, NLP and computer vision tasks such as named entity recognition, text classification, object detection, image segmentation, evaluation and more.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/5/5461f212355e97d2350e43a264e66635ef89b95c.jpeg)

One of the motivations for this is `partial_overlap`, where you may specify that all annotators review x% of data. That data may be used for calibration like inter-annnotator agreement. Alternatively, you may set up routes based on different criteria -- one being the flagged information mentioned previously. With a little trial-and-error, I bet you could build a sophisticated routing system that may have multiple stages.

Just curious - while you have no internet, would it be possible to run Docker? This may help you in running instances that could even be possible scaled up (say Kubernetes) on premise.

We also wrote new deployment docs where we provide ideas of how to deploy Prodigy via Docker (among other ways):

> **[Deploying Prodigy · Prodigy · An annotation tool for AI, Machine Learning...](https://future.prodi.gy/docs/deployment/)**
>
> A downloadable annotation tool for LLMs, NLP and computer vision tasks such as named entity recognition, text classification, object detection, image segmentation, evaluation and more.

> [@sudarshan85](#):
>
> calculate the interrator agreement and other metrics

We're also planning to release soon new "Metrics" components that include built-in and custom metrics like IAA. We decided to move it as a follow-up to our initial v1.12.0 release but I can post back once that's available (either thru alpha or released). One benefit of having IAA integrated with Prodigy would be to compute the metrics on the fly - as a means of ongoing annotation QA, which could make task routing even more powerful!

> [@sudarshan85](#):
>
> rule-based model that analyzes clinical notes

In v1.12, check out the new [`filter_by_patterns` recipe](https://future.prodi.gy/docs/recipes#filter-by-patterns). Perhaps it can help 🙂

That's my initial thoughts but the team can look over your post next week and see if there are additional suggestions.

Hope this helps!

---

<div class="post-metadata">

**Author:** ![sudarshan85](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sudarshan85/32/2199_2.png) [@sudarshan85](https://support.prodi.gy/u/sudarshan85)\
**Post date:** [July 6, 2023, 7:33pm UTC](https://support.prodi.gy/t/how-to-manage-multiple-annotators/6643/3 "2023-07-06T19:33:17Z")

</div>

Hi @ryanwesslen,

I've been out for the past week and will start working on this again next week. Thank you for your kind words about what we've done!

I started reading your reply and stoped and just scrolled and was amazed at the lengths and scale of the reply. I felt that I should post a reply and thank you for the effort. I will read this detailed reply and update the post based on my progress.

Thank you!

---

<div class="post-metadata">

**Author:** ![sudarshan85](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/sudarshan85/32/2199_2.png) [@sudarshan85](https://support.prodi.gy/u/sudarshan85)\
**Post date:** [July 12, 2023, 7:18pm UTC](https://support.prodi.gy/t/how-to-manage-multiple-annotators/6643/4 "2023-07-12T19:18:04Z")

</div>

> [@ryanwesslen](#):
>
> Use the flagged annotations to other annotators

I think this is the easiest method right now. I'm going to try it out and get feedback from the SMEs and see how it goes.

> [@ryanwesslen](#):
>
> Just curious - are you aware of the difference between your global and local `prodigy.json` and overrides?

I am but I decided to write out a global json file for each user because the code base which runs the script is shared. The SMEs don't really like to use git commands to clone repos. Since its shared and each configuration is different, I can't place it in the local code directory.

> [@ryanwesslen](#):
>
> Just curious - while you have no internet, would it be possible to run Docker? This may help you in running instances that could even be possible scaled up (say Kubernetes) on premise.

The admins don't like using docker. I understand that might not be a satisfying answer 🙂 but my hands are tied with respect to the installs. Its hard but thats the environment that I have to work in.

Thanks.