Starting a Multi-user session

[ryanwesslen]

@NoorKhalifa

While reading, I saw that the current version of Prodigy is fine for multi-user annotations using the same port, right?

Correct. An alternative to multi-user named sessions is to run each annotator on a different port/dataset.

Here's a good pro/con of each:

Help - Multiple Annotators setup

Ah, sorry if the terminology I used was confusing! What I meant was the session/instance/process. There are generally two ways you could set this up and it depends on your project, annotators and how you want to set up the workflow: Start one Prodigy process one a given host/port per annotator and save the annotations to separate datasets. For example, on port 8080, you start the Prodigy server with 100 documents and save to annotations_andrew, on port 8081 you start Prodigy with the next 100 …

@NoorKhalifa

I will be able to allow each annotator to annotate the whole dataset and I will be able to get each annotated dataset separately, right?

Sounds like then you want "overlapping" annotations, i.e., you need to set feed_overlap to true. You can do this in prodigy.json or as an override.

I just responded to a similar example; see the bottom of the difference of feed_overlap:

NER Team

hi @kushalrsharma, When all sessions have the same data, we call that "overlapping" annotations. However, it seems that you want "non-overlapping" annotations, i.e., each example in the data once to whoever is available. As a result, your data will have each example labelled only once in total. This is exactly what setting "feed_overlap" to false will do (this is the value by default). But it seems like you're saying that you think your "feed_overlap" is still set to false, but it is not pr…

@NoorKhalifa

However, when I save the first 10 then go on to annotate more records and exit or refresh, the new progress is still not being saved. Is there a way of enforcing autosave?

If you want to autosave, you can either set your batch_size to 1 or setting instant_submit in prodigy.json (config) to true. The one downside of these approaches is that it'll remove the undo as you'll no longer be holding the batch in client before sending to DB.

I can't remember the specific differences but this post seems to discuss it more:

Synchronous Batch Mode

Hi! One of the main concerns for the web app is to make sure the queue is never running low, so it will keep asking for new batches in the background as the current batch is being annotated. The motivation here is that one individual example typically doesn't matter very much, and a single update to the model won't move the needle enough for it to be worth blocking on it. Depending on the model, blocking could easily take 10+ seconds or even a minute, so that's not really viable to do synchronou…

Also, the Progress Bar is not updated real time; it's only updated when a new batch is retrieved. We've been debating modifying this in a future version but there are some unintended consequences in high latency environments with multiple annotators that can cause issues.

Blocks and Progress bar

Hi! Prodigy will calculate the progress automatically if the stream returned by the recipe has a length (e.g. if it's a list). If the stream is a generator, it could potentially be infinite and it will only be read one batch at a time, so Prodigy can't know how many examples are left. If you know how many examples you have, or want to implement some other custom logic to calculate the progress, you can also add a "progress" callback to the components returned by your recipe: https://prodi.gy/d…

If you wanted something more realtime, you could check out the update callback. @koaning has a great video on it to track annotator speed:

Using the Update callback - Prodigy Shorts

Since, you're using feed_overlap is true, be sure to update Prodigy today. Yesterday, we released v1.11.9 which fixed a lingering bug of "duplicated" annotations in high latency/multiple annotators sessions.

Also, it's worth mentioning the team is working very hard on releasing Prodigy v2 in a few months. This will add even greater customization to feed_overlap (e.g., what if you want only a certain percentage overlapping across annotators).

For the release of Prodigy v2, we have some exciting new features and a significant redesign to the way examples are sent to different annotators. Specifically, this redesign will eliminate the need for this tradeoff and should eliminate 100% of unwanted duplicates.

It will also bring more customization to the feed_overlap setting like setting the number of annotations you want for each task , or configuring a percentage of examples to have overlapping annotations for. We're even working on registering custom policies to distribute work to different annotators.

Also, since you're looking at having multiple annotators, be sure to check out @pmbaumgartner great Inter-annotator Agreement recipes. This can allow you to better "calibrate" how consistent annotators are (see Peter's wonderful NormConf talk on why calibration is important). We'd love feedback as this project is evolving!

Inter-Annotator Agreement Recipes - Feedback Wanted

Hey everyone - I thought I would share some recipes I made for calculating Inter-Annotator Agreement (also called Inter-Rater Reliability) in Prodigy. It supports a variety of annotation situations so you can have any number of annotators and incomplete overlap of annotations. Currently it supports classification tasks (binary/multiclass/multilabel), but not any spans, NER, audio, or image tasks. Here's a link to the repo: GitHub - pmbaumgartner/prodigy-iaa Everything is bundled as a …

Thanks again for your questions (sorry about dumping a ton of resources). Hopefully you have plenty of resources and please post back if you have interesting experiments or link to blog/paper if you're successful!