NER Team

[kushalrsharma (Kushu)]

Hello,
I and my team are currently utilizing my company’s license for Prodigy to annotate some data for NER i have like more than 5 members for annotating that I am interested in later analyzing with Spacy. As part of the annotation process, is there a way to resume annotation where you left off from your previous session and start again.
Please advise.

[ryanwesslen]

hi @kushalrsharma!

Thanks for your question.

@kushalrsharma

As part of the annotation process, is there a way to resume annotation where you left off from your previous session and start again.

By default, Prodigy should resume annotation based on where you left off. So I'm not sure on why you don't think that is the case.

Since you have multiple annotators, have you changed feed_overlap?

The "feed_overlap" setting in your prodigy.json or recipe config lets you configure how examples should be sent out across multiple sessions. If true , each example in the dataset will be sent out once for each session , so you’ll end up with overlapping annotations (e.g. one per example per annotator). Setting "feed_overlap" to false will send out each example in the data once to whoever is available. As a result, your data will have each example labelled only once in total.

This setting will change the behavior of how annotations are sent out across multiple annotators. By default, feed_overlap is false, so where Prodigy starts after stopping the server will depend on what was the last annotation saved to the database (be sure that annotators click the "save" button when they're done so that the last batch is saved to the database).

[kushalrsharma (Kushu)]

Is this wierd?

export PRODIGY_ALLOWED_SESSIONS=Simran,Saru,Anupam,Heera,Kshitiz,Nirajan,Samikshya,Saurav,Prakash 

!python -m prodigy spans.manual span_resume  blank:en /home/kushal/Documents/spacyprodigy/Prakash/jsonlfiles/resume365.jsonl --label PROFILE,PROFILE_NAME,PROFILE_EMAIL,PROFILE_ADDRESS,PROFILE_SOCIAL,PROFILE_SUMMARY,EXPERIENCE,EXPERIENCE_ORG,EXPERIENCE_ADDRESS,EXPERIENCE_STARTDATE,EXPERIENCE_ENDDATE,EXPERIENCE_SKILLS,EXPERIENCE_KNOWLEDGE,ACADEMICS,ACADEMIC_INSTITUTIONS,ACADEMIC_ADDRESS,ACADEMIC_STARTDATE,ACADEMIC_ENDDATE,ACADEMIC_GPA,ACADEMIC_COURSES

When i use http://localhost:8080/?session=Nirajan, http://localhost:8080/?session=Saru
The data that are shown across session are same. why?

prodigy.json 
{
	"feed_overlap": false
}

What can be done to have the different data among different sessions? Is it due to this >/home/kushal/Documents/spacyprodigy/Prakash/jsonlfiles/resume365.jsonl since it consists of only data?

[ryanwesslen]

hi @kushalrsharma,

@kushalrsharma

The data that are shown across session are same. why?

When all sessions have the same data, we call that "overlapping" annotations.

However, it seems that you want "non-overlapping" annotations, i.e., each example in the data once to whoever is available. As a result, your data will have each example labelled only once in total.

This is exactly what setting "feed_overlap" to false will do (this is the value by default).

But it seems like you're saying that you think your "feed_overlap" is still set to false, but it is not producing that behavior, right?

Try to run:

PRODIGY_CONFIG_OVERRIDES='{"feed_overlap": false}' python -m prodigy ...

This should override everything. I'm concerned you may have accidentally set "feed_overlap": true at some point or not pointing to the correct prodigy.json. You can technically have a prodigy.json for your project and one for global. (You may want to run python -m prodigy stats to verify what path your global prodigy.json is).

Also, you may want to consider reseting your overrides:

export PRODIGY_CONFIG_OVERRIDES="{}"

PRODIGY_CONFIG_OVERRIDES: issue

Hi! Since you've used export, the environment variable has been set for the whole session, so one option would be to just set export PRODIGY_CONFIG_OVERRIDES="{}" to reset it to an empty dict. I'm not sure if this happened while copy-pasting or if this was the actual output, but based on the error message, it looks like you somehow ended up with typographic quotes, e.g. “ instead of "? So that's the first thing I would check. If you want to be extra sure, you can also run echo $PRODIGY_CONFIG_O…

Last, when providing examples, please provide reproducible examples for us. This will help you get faster responses. Since we don't have your data, it is impossible for us to help you to confirm what the problem and that we're talking about the same issue. I've created this example below that shows this to help you see what should be the difference.

Using this data:
nyt_text_dedup.jsonl (18.5 KB)

feed_overlap: false ("non-overlapping")

PRODIGY_CONFIG_OVERRIDES='{"feed_overlap": false}' python3 -m prodigy ner.manual ner_ex blank:en nyt_text_dedup.jsonl --label ORG

First, open browser for session1: "ryan"

Don't annotate any examples.

Then open a 2nd browser for session2: "kushu"

Notice how this starts "kushu" at record number 10 (since batch_size is 10).

feed_overlap: true (overlapping)

PRODIGY_CONFIG_OVERRIDES='{"feed_overlap": true}' python3 -m prodigy ner.manual ner_ex blank:en nyt_text_dedup.jsonl --label ORG

First, open browser for session1: "ryan"

Don't annotate any examples.

Then open a 2nd browser for session2: "kushu"

Notice how this starts "kushu" at record number 0 (that is, "ryan" and "kushu" have the same data, hence their annotations "overlap").

Hope this helps to clarify and let me know if this clears up any confusion!

[kushal_pythonist (Kushal Raj Sharma)]

Thank You @ryanwesslen it worked.

[kushalrsharma (Kushu)]

@ryanwesslen do this works in case of different users connected to the same wifi? Extremely needed this

[ryanwesslen]

hi @kushalrsharma!

@kushalrsharma

do this works in case of different users connected to the same wifi?

I think the answer is yes. I'm a bit confused on what you mean by "connected to the same wifi". Typically, they need access to the server that hosts your Prodigy session and that host was modified (e.g., 0.0.0.0).

Serving on other than localhost

Hi! I’d like to run the annotation UI on our local server and access it without ssh tunneling. For this it would be nice if I can set the serving ip other than “localhost” (e.g “0.0.0.0”). I could not find nor a "host" nor an "ip" option in the readme, but luckily this just worked fine: {"host": "0.0.0.0"} I’d suggest to document this in the readme. /Gyuri

User authentication for Prodigy web app

Setting the host to “0.0.0.0” (any IP on the machine) solved it! Thanks a lot, @honnibal

But please remember -- there are many challenges in setting up annotators to access from external computers like firewall, reverse proxy for setting up https, authentication, etc. Here are just a few examples:

Access Prodigy from external computer

Dear All, Really happy with Prodigy. Until now, due to our requirements, we had all annotators annotate locally. However now we try to setup a server such that multiple annotators can code on one instance. We setup prodigy on Ubuntu 20.04, no other webservices are running. A) python3 -m prodigy ner.manual etc…………. We can access prodigy on localhost:8080 We can access prodigy on 127.0.0.1:8080 We are not able to reach Prodigy from an external computer. We added rules to the firewall: …

Prodigy with https

I would recommend adding a reverse proxy in front that is restricted to only https connections. For instance, you would advertise the service on localhost, and have something like Nginx, Traefik or Apache communicating externally, via https.

authentication out of the box

You can set up the basic auth using the environment variables PRODIGY_BASIC_AUTH_USER and PRODIGY_BASIC_AUTH_PASS. See here for an example: If you want a fully custom solution, you could also add you own authentication miniservice that asks the user to log in and then starts up the server and redirects them. You can find different approaches for this on the forum.

My main takeaway is please remember to keep searching for past tickets. While we'll do our best to help, this is a non-trivial problem to scale up many annotators. We have found many users can create such processes on their own, but they need to have good skills in managing networks/security and it takes time to develop these.