Starting a Multi-user session

[NoorKhalifa]

I read all the docs and support cases but I did not manage to find a step by step tutorial for starting a multi-user session.

I set up the session on my local host and I managed to enter separate sessions by adding ?session=noor but I can only do that on my laptop

But how do I send this to someone who does not use my laptop and internet connection. My annotators are not in the same place/company.

[ryanwesslen]

hi @NoorKhalifa!

Thanks for your question and welcome to the Prodigy community

@NoorKhalifa

But how do I send this to someone who does not use my laptop and internet connection. My annotators are not in the same place/company.

Please see this post:

Creating a web server and share URL with other annotators

Hi! There are lots of ways to do this and it kinda depends on what you need. The simplest "elegant" solution is to run Prodigy on a remote machine/server on a given port and open up that port. You can then send your annotators the URL of your remote machine. If you want this to be nicer, you can add a domain to it. The simplest "quick and dirty" solution would be to use a service like ngrok or localtunnel to expose your localhost via a public URL. This means you can run Prodigy on your local ma…

This post too provides more details on ngrok:

Tip: Making a local Prodigy service externally accessible with ngrok.com

Sometimes you want the server running on your dev machine to be accessible to someone else — or even just yourself, so you can do some annotation on the train or from the couch with your mobile device. I haven’t tried it yet, but this service looks like it should be useful: https://ngrok.com Has anyone used this for other projects? They have a free tier, and the full service isn’t very expensive. If you try it out, let us know how it goes!

But as mentioned, there are a lot of ways this can be done and really depends on your setup (e.g., on-premise vs cloud, which cloud provider, security/firewall requirements, reverse proxy/load balancer). This is why there isn't a simple step-by-step tutorial. I would recommend checking out more of the multi-user tagged posts that have many examples.

The team is working very hard on Prodigy Teams that provides this functionality:

Prodigy Annotation Manager Update // Prodigy Scale // Prodigy Teams

Many of you have been asking about how to scale your Prodigy projects – including how to manage more annotators, how to keep multiple feeds running, and how to make sure your annotators stay in agreement. These are challenging problems that are outside of the scope of the Prodigy software itself, so we've always planned an extension product that helps you use Prodigy in larger products. We've been quietly working away on this for some time, so we're excited to finally announce our progress! The …

We'll post updates once it becomes publicly available.

[NoorKhalifa]

Thanks Ryan!

I managed to connect the Prodigy session on my phone and other devices via Ngrok.

Is there a way to track each annotator's progress (number of annotated records)?

Also, currently, when I refresh the Ngrok link, the session's and the total's progress changes to 0% as if no annotations were done. Is it really deleting previous progress? If so, how do I prevent that?

Lastly, from where can I access the annotated dataset by each annotator?

[ryanwesslen]

@NoorKhalifa

Is there a way to track each annotator's progress (number of annotated records)?

Check out this:

Find out annotator progress

Hi! Are you storing anything with your individual examples that indicates the document the text belongs to (or the line number, a running index, something like that)? Ultimately, it depends on how you define "progress" and the metrics you're looking for, but let's say your examples look like this and include the number of the document and a running ID (e.g. nth sentence in the document): {"text": "...", "document_no": 5, "id": 1234} You could then, for instance, do something like this to find …

Now it separates by metadata (document); but you could modify for _session_id. I really like the idea from the post of creating a streamlit app like this (FYI, this uses a different approach for tracking annotators by giving each annotator their own port/Prodigy dataset.).

Also you may want to check out the progress recipe:

prodigy progress news_headlines_person,news_headlines_org
================================== Legend ==================================

New         New annotations collected in interval
Total       Total annotations collected
Unique      Unique examples (not counting multiple annotations of same example)
================================= Progress =================================

               New   Unique   Total   Unique
-----------   ----   ------   -----   ------
10 Jul 2021   1123      733    1123      733
12 Jul 2021    200      200    1323      933
13 Jul 2021    831      711    2154     1644
14 Jul 2021    157      150    2311     1790
15 Jul 2021   1464     1401    3775     3191

It doesn't break down by "session_id"; however, you can view the underlying recipe and modify it (e.g., replace time with "session_id" when it prints out the table (or something like it). To find the location of the recipe, run python -m prodigy stats and find the Location: path. Open that folder, then look for /recipes/commands.py. You can then use that as a custom recipe and run it with the -F new_recipe.py. If you get either to work, please post it back so other members of the community can use it!

@NoorKhalifa

Also, currently, when I refresh the Ngrok link, the session's and the total's progress changes to 0% as if no annotations were done.

Did you annotate more than 10 examples and/or make sure to click "Save"? By default, the first 10 example (as batch_size is 10 by default) will not be saved to the database unless you save it or get through those first 10 (then it'll automatically save, and retrieve a new batch).

Are you using named multi-user sessions? You likely would want to as there's no other way to identify your data by annotator. Also, be sure to be aware of the difference of feed_overlap (that is, do you want overlapping or non-overlapping annotations.

@NoorKhalifa

Is it really deleting previous progress? If so, how do I prevent that?

No. If you saved the annotations to the database, then it's not "deleting" anything. I suspect your problem was that the annotations were made (less than 10), but you didn't save them to DB by clicking save.

@NoorKhalifa

Lastly, from where can I access the annotated dataset by each annotator?

Yes, use get_dataset and filter by "_session_id".

from prodigy.components.db import connect

db = connect()
examples = db.get_dataset("my_dataset")

# filter examples by `"session_id"`

This is similar to the earlier post.

Since you're annotating on your phone, you may also like @koaning's Prodigy Short on tips on running Prodigy on a mobile device:

Swipe and Annotate on Phones - Prodigy Shorts

[NoorKhalifa]

@ryanwesslen

this uses a different approach for tracking annotators by giving each annotator their own port/Prodigy dataset

While reading, I saw that the current version of Prodigy is fine for multi-user annotations using the same port, right? I will be able to allow each annotator to annotate the whole dataset and I will be able to get each annotated dataset separately, right?

@ryanwesslen

FYI, this uses a different approach for tracking annotators by giving each annotator their own port/Prodigy dataset.

Is this do-able for the single port approach?

@ryanwesslen

Did you annotate more than 10 examples and/or make sure to click "Save"?

You were right, I did not save the progress. However, when I save the first 10 then go on to annotate more records and exit or refresh, the new progress is still not being saved. Is there a way of enforcing autosave?

@ryanwesslen

Are you using named multi-user sessions?

Yes I am. I will make sure to change the settings of feed_overlap as I want each annotator to annotate the whole dataset.

Thanks again, Ryan. I really appreciate the detailed answers. You make this platform extremely beginner-friendly.

[ryanwesslen]

@NoorKhalifa

While reading, I saw that the current version of Prodigy is fine for multi-user annotations using the same port, right?

Correct. An alternative to multi-user named sessions is to run each annotator on a different port/dataset.

Here's a good pro/con of each:

Help - Multiple Annotators setup

Ah, sorry if the terminology I used was confusing! What I meant was the session/instance/process. There are generally two ways you could set this up and it depends on your project, annotators and how you want to set up the workflow: Start one Prodigy process one a given host/port per annotator and save the annotations to separate datasets. For example, on port 8080, you start the Prodigy server with 100 documents and save to annotations_andrew, on port 8081 you start Prodigy with the next 100 …

@NoorKhalifa

I will be able to allow each annotator to annotate the whole dataset and I will be able to get each annotated dataset separately, right?

Sounds like then you want "overlapping" annotations, i.e., you need to set feed_overlap to true. You can do this in prodigy.json or as an override.

I just responded to a similar example; see the bottom of the difference of feed_overlap:

NER Team

hi @kushalrsharma, When all sessions have the same data, we call that "overlapping" annotations. However, it seems that you want "non-overlapping" annotations, i.e., each example in the data once to whoever is available. As a result, your data will have each example labelled only once in total. This is exactly what setting "feed_overlap" to false will do (this is the value by default). But it seems like you're saying that you think your "feed_overlap" is still set to false, but it is not pr…

@NoorKhalifa

However, when I save the first 10 then go on to annotate more records and exit or refresh, the new progress is still not being saved. Is there a way of enforcing autosave?

If you want to autosave, you can either set your batch_size to 1 or setting instant_submit in prodigy.json (config) to true. The one downside of these approaches is that it'll remove the undo as you'll no longer be holding the batch in client before sending to DB.

I can't remember the specific differences but this post seems to discuss it more:

Synchronous Batch Mode

Hi! One of the main concerns for the web app is to make sure the queue is never running low, so it will keep asking for new batches in the background as the current batch is being annotated. The motivation here is that one individual example typically doesn't matter very much, and a single update to the model won't move the needle enough for it to be worth blocking on it. Depending on the model, blocking could easily take 10+ seconds or even a minute, so that's not really viable to do synchronou…

Also, the Progress Bar is not updated real time; it's only updated when a new batch is retrieved. We've been debating modifying this in a future version but there are some unintended consequences in high latency environments with multiple annotators that can cause issues.

Blocks and Progress bar

Hi! Prodigy will calculate the progress automatically if the stream returned by the recipe has a length (e.g. if it's a list). If the stream is a generator, it could potentially be infinite and it will only be read one batch at a time, so Prodigy can't know how many examples are left. If you know how many examples you have, or want to implement some other custom logic to calculate the progress, you can also add a "progress" callback to the components returned by your recipe: https://prodi.gy/d…

If you wanted something more realtime, you could check out the update callback. @koaning has a great video on it to track annotator speed:

Using the Update callback - Prodigy Shorts

Since, you're using feed_overlap is true, be sure to update Prodigy today. Yesterday, we released v1.11.9 which fixed a lingering bug of "duplicated" annotations in high latency/multiple annotators sessions.

Also, it's worth mentioning the team is working very hard on releasing Prodigy v2 in a few months. This will add even greater customization to feed_overlap (e.g., what if you want only a certain percentage overlapping across annotators).

For the release of Prodigy v2, we have some exciting new features and a significant redesign to the way examples are sent to different annotators. Specifically, this redesign will eliminate the need for this tradeoff and should eliminate 100% of unwanted duplicates.

It will also bring more customization to the feed_overlap setting like setting the number of annotations you want for each task , or configuring a percentage of examples to have overlapping annotations for. We're even working on registering custom policies to distribute work to different annotators.

Also, since you're looking at having multiple annotators, be sure to check out @pmbaumgartner great Inter-annotator Agreement recipes. This can allow you to better "calibrate" how consistent annotators are (see Peter's wonderful NormConf talk on why calibration is important). We'd love feedback as this project is evolving!

Inter-Annotator Agreement Recipes - Feedback Wanted

Hey everyone - I thought I would share some recipes I made for calculating Inter-Annotator Agreement (also called Inter-Rater Reliability) in Prodigy. It supports a variety of annotation situations so you can have any number of annotators and incomplete overlap of annotations. Currently it supports classification tasks (binary/multiclass/multilabel), but not any spans, NER, audio, or image tasks. Here's a link to the repo: GitHub - pmbaumgartner/prodigy-iaa Everything is bundled as a …

Thanks again for your questions (sorry about dumping a ton of resources). Hopefully you have plenty of resources and please post back if you have interesting experiments or link to blog/paper if you're successful!

[NoorKhalifa]

Thanks again, Ryan! I will refer to all the resources and see what works for me. I appreciate the resources provided and will definitely post back once I'm done with my project

[kushal_pythonist (Kushal Raj Sharma)]

@NER Team

First, open browser for session1: "ryan"

@NER Team

Then open a 2nd browser for session2: "kushu"

This works perfectly fine on local host, but not on ngrok platform

[ryanwesslen]

I haven't used ngrok extensively (i.e., beyond only having one annotator). You may need to research it to find how you can pass the session name (if it's possible). Remember, ngrok is a "quick-and-dirty" tool. It is by no means a perfect solution, especially when handling many annotators. You may also want to try localtunnel as an alternative.

Creating a web server and share URL with other annotators

Hi! There are lots of ways to do this and it kinda depends on what you need. The simplest "elegant" solution is to run Prodigy on a remote machine/server on a given port and open up that port. You can then send your annotators the URL of your remote machine. If you want this to be nicer, you can add a domain to it. The simplest "quick and dirty" solution would be to use a service like ngrok or localtunnel to expose your localhost via a public URL. This means you can run Prodigy on your local ma…

[kushal_pythonist (Kushal Raj Sharma)]

this are the sessions i have created running smoothly in localhost but when i use ngrok and expose the ports and share to different people for annotating the documents are same . Why?

[ryanwesslen]

hi @kushal_pythonist!

@kushal_pythonist

when i use ngrok and expose the ports and share to different people for annotating the documents are same . Why?

I haven't used ngrok beyond one annotator. ngrok is a third-party tool so we don't offer support. Can you check out the ngrok documentation to find out?

[NoorKhalifa]

@ryanwesslen

Yes, use get_dataset and filter by "_session_id".

I am running my Prodigy session on a Raspberry Pi as a server. Can I access the annotated datasets using Python in my Windows Machine?

from prodigy.components.db import connect
db = connect()
examples = db.get_dataset_examples("my_dataset")

I ran this code, but the variable examples does not show anything

[ryanwesslen]

Are you 100% sure you have the right name of the dataset and/or that there's actually data that dataset?

Let's assume you saved data into a Prodigy dataset called my_dataset.

Can you run?

prodigy print-dataset my_dataset

Or another alternative is:

python -m prodigy db-out my_dataset > my_dataset.jsonl

I don't think running Python on a windows machine would be a problem. Just curious, can you run:

print(db.datasets)

and do you see my_dataset (or the name of the dataset you're looking for)?

I did notice that if you run db.get_dataset_examples("my_dataset") and you don't have a dataset named my_dataset, it won't print a warning that "Data set doesn't exist".

[NoorKhalifa]

@ryanwesslen

and do you see my_dataset (or the name of the dataset you're looking for)?

I ran

print(db.datasets)

and got the list

['ner_news_headlines', 'comment_class', 'news_topic', 'news_topics']

the current dataset is not included in the list. Does the Prodigy session need to end so that it saves or can I access the dataset while running? Because my session is still running since some annotators still did not finish

[ryanwesslen]

@NoorKhalifa

Does the Prodigy session need to end so that it saves or can I access the dataset while running? Because my session is still running since some annotators still did not finish

No, you don't need to end your session. Annotations are saved to the database after each batch. That is, click the "save" button or complete 1 batch, which by default is 10 annotations.

For example, run a recipe into a new dataset. Annotate 10 examples, but don't save or go to the 11th example (if you still have default batch size 10). Then try to look records in that dataset. You should likely not see anything. However, click the save button (or proceed). Now try to look at your data again. You should see your examples. If you don't then you may not be looking at the correct dataset.

Also, consider adding in logging when running your recipes. When you do this, you should see your examples when they're sent to the database (let alone, it'll confirm the name of your database).

[NoorKhalifa]

I ran

python -m prodigy db-out my_dataset > my_dataset.jsonl

on my Raspberry Pi and it worked. I believe that the command must be run in the device that initially ran the Prodigy session.

It would be great if that is not the case and the dataset can be accessed from anywhere.