Working collaboratively with Prodigy

[kushal_pythonist (Kushal Raj Sharma)]

I purchased the licensed(more than 5) to work on NER Project. I do not have any idea how i work collaboratively using these licensed prodigy account. I just want to know how i go with the process. I have 5 peoples to work with me. What and how should i perform the task. Can anyone please explain the process and help me out to complete the project?

[ryanwesslen]

hi @kushal_pythonist!

Thanks for your question.

@kushal_pythonist

I purchased the licensed(more than 5) to work on NER Project. I do not have any idea how i work collaboratively using these licensed prodigy account. I just want to know how i go with the process. I have 5 peoples to work with me.

First, it's important to determine the "what" you'll be labeling:

How to use Prodigy for NER labelling as a team?

Hi! There's no one correct answer and how you structure your projects depends on your specific use case. When you're starting a new project that involves training a model and collecting new data, you'll usually have two distinct phases: Development phase. During this phase, you'd validate ideas for the model and it typically requires the data scientists/developers working very closely with the annotators, and often even annotating small samples themselves to test what works best. Even if the …

@kushal_pythonist

Can anyone please explain the process and help me out to complete the project?

Here's more of the "how" to implement your project.

First, if your annotators are on the same network (e.g., server), check out our documentation on named multi-user sessions. You can start a Prodigy session as you would normally:

python prodigy ner.manual ner_news_headlines blank:en ./news_headlines.jsonl --label PERSON,ORG,PRODUCT,LOCATION

✨ Starting the web server at http://localhost:8080 ...
Open the app in your browser and start annotating!

To create a custom named session, add ?session=xxx to the annotation app URL. For example, annotator Alex may access a running Prodigy project via http://localhost:8080/?session=alex. Internally, this will request and send back annotations with a session identifier consisting of the current dataset name and the session ID – for example, ner_person-alex. Every time annotator Alex labels examples for this dataset, their annotations will be associated with this session identifier.

The "feed_overlap" setting in your prodigy.json or recipe config lets you configure how examples should be sent out across multiple sessions. If true, each example in the dataset will be sent out once for each session, so you’ll end up with overlapping annotations (e.g. one per example per annotator). Setting "feed_overlap" to false will send out each example in the data once to whoever is available. As a result, your data will have each example labelled only once in total.

As of v1.8.0, the PRODIGY_ALLOWED_SESSIONS environment variable lets you define comma-separated string names of sessions that are allowed to be set via the app. For instance, PRODIGY_ALLOWED_SESSIONS=alex,jo would only allow ?session=alex and ?session=jo, and other parameters would raise an error.

An alternative approach is where you have one session with a unique port and dataset for each annotator:

Multiple user sessions for multiple datasets

Hi! The named multi-user sessions are really mostly intended for workflows where you want to send out the same data to different annotators, based on who is available. If you already know which input source should be annotated by a session, and you want every session to annotate different data, you should just start multiple separate instances for the different users. You can do this by setting the PRODIGY_PORT environment variable and running Prodigy on different ports. Each user can then have …

This is harder to manage manually when you go beyond a few annotators. Some community members have found using tools like tmux to help:

How are you managing multiple projects?

I keep it simple. I have a document with 20+ command lines of the form: PRODIGY_CONFIG=custom_config.jsonl PRODIGY_PORT=8251 PRODIGY_BASIC_AUTH_USER=user PRODIGY_BASIC_AUTH_PASS=password prodigy --all_the_options This is used so that when I have to add more documents for annotation/change to a model in the loop/update whatever I can easily do that. It also helps me keep track of the ports. The Prodigy instances are started up in a Tmux session.... Yeah, barbarian method, but oh well, it's been…

Second, if you're running on this on a different machine than what your annotators are on, you'll need to modify the host, changing Prodigy Host to "0.0.0.0" from localhost.

To do this, you'll need to modify the configuration in the prodigy.json file or setting the environment variable for PRODIGY_HOST.

Starting Prodigy server on a remote machine ('--no-browser --port=XXXX)

Try setting the environment variable export PRODIGY_HOST="0.0.0.0" before starting the Prodigy server. This should advertise the server on the WAN IP, instead of the loopback interface (localhost, which is the default). You can also configure the port with the PRODIGY_PORT environment variable. For production setups, you might want to look into using a reverse proxy in front of the service. I like to use Traefik. Other popular choices include Apache, Nginx and Caddy.

As mentioned, for production setups, you may want to consider reverse proxies. Also, be aware that you may have to handle firewalls or other issues depending on your server setup.

Access Prodigy from external computer

Dear All, Really happy with Prodigy. Until now, due to our requirements, we had all annotators annotate locally. However now we try to setup a server such that multiple annotators can code on one instance. We setup prodigy on Ubuntu 20.04, no other webservices are running. A) python3 -m prodigy ner.manual etc…………. We can access prodigy on localhost:8080 We can access prodigy on 127.0.0.1:8080 We are not able to reach Prodigy from an external computer. We added rules to the firewall: …

Developing comprehensive guidelines for working collaboratively is hard, because a lot of design decisions depend on the unique network setup. There are many server posts. and cloud posts like aws or google-cloud that may help depending on your setup.

Hope this helps!

[kushal_pythonist (Kushal Raj Sharma)]

Lets take a example here:
I want to annotate NER. I have 5 poeples streamlined
I use this command :

prodigy ner.manual resume blank:en /home/random/Documents/anno/outputfiles/randompersonresume.jsonl --label PERSON,LOCATION,UNIVERSITY,ROLE,ORGANIZATION,DATE,DOMAIN,PROJECT,EXPERIENCE,HARDSKILLS,SOFTSKILLS,DEGREE

to annotate. In first day, they complete the annotations. In second day is this the same way to start over? Or any other way?
I know this may be silly questions but i need to solve this issue asap. Thank You. Appreciated

[ryanwesslen]

hi @kushal_pythonist!

Unfortunately, the answer is "it depends" on your use case and Ines' post is the best advice for how to start your project.

A lot of the decisions you need to make is based on what your use case is. Matt discussed this as the base of the "ML Hierarchy of Needs":

Building new NLP solutions with spaCy and Prodigy - Matthew Honnibal

I suspect you're in a rush so likely may not have time to step back and focus on these problems.

If you need something, the closest step-by-step instructions I can provide is our NER flowchart:

Prodigy_NER_flowchart_v2_0_0_light.pdf

Hope this helps!

[kushal_pythonist (Kushal Raj Sharma)]

Thank you so much @ryanwesslen . I will surely watch the video you provided. And thanks again for your constant help and guidance.