add new lables as per new data received to existing data set and retrain the NER model

[Vishal112]

Pls guide me regarding this

[ryanwesslen]

hi @Vishal112!

That's a bit of a tricky question.

One of the first questions you should ask isn't really a ML question but a business question: what's the expectation on the frequency/timing for new NER labels? and when changed, how are annotation guidelines updated so that your annotators have clear definitions of the new labels?

If you have a model in production, I would caution against an expectation that you can add/retrain many times and on an ad hoc (non-regular occuring) basis (it would be okay if you're only in model development developing the model though). The reason is it may be very difficult to track changes so you likely should agree with your model users (stakeholders) on fixed times to add new labels (e.g., once a month).

Along those same lines, what's important (and sometimes forgotten) is to ensure you have a clear definition of what you're labeling through explicit annotation guidelines. This can be as simple as definitions of your entities. One great example is from the Guardian, who published an article about how they used Prodigy for a ner model:

Talking sense: using machine learning to understand quotes

They published their code and their annotation guidelines. As they discussed, it's important to have routine discussions with your modelers and business stakeholders to constantly update those guidelines.

The best example of this philosophy is in Matt's 2018 talk where he talks about successful ways of defining the business problem, the need for clear annotation guidelines, and role the that an iterative approach can aid in resulting in successful vs. unsuccessful ML projects:

Building new NLP solutions with spaCy and Prodigy - Matthew Honnibal

Now I suspect you're more interested in how to implement updating NER labels and retrain the model, there are a lot of past Prodigy Support issues and documentation that can help.

• First, I'd start with our NER workflow. We're hoping very soon to update this as some of the Prodigy recipe names have changed (e.g., train now instead of batch.train). But the main idea still holds. An important part is whether you're using a pre-trained model (e.g., en_core_web_sm) to update and/or add new existing entity types or starting a new model. What's important is that we recommend if you're adding more than 3+ new entity types, you're likely better off starting from scratch.

• If you have some prior knowledge about your new entity (e.g., terms that are related), you should also consider adding match patterns to help bootstrap your entities. This will make it easier to label as these patterns will come highlighted.

• If you are starting from scratch and simply want to create a workflow for a few new entities, I like my teammate's @ljvmiranda921 advice in this post that is very similar to your question:

Adding new label

Hi @pkras! Is there a way to use ner.manual or ner.correct on the saved labelled dataset and add the new label? It's possible to load existing datasets again. Prodigy will load the anotations from the dataset then stream them back in. Perhaps you can try that then include your new label: prodigy ner.manual <new_dataset> <model> dataset:<old_dataset> --label PERSON,ORG Similarly, i would like to ask what you would advise in the case of wanting to remove a label from the existing scheme. D…

• It's also good to be aware that there are ways to create "nested" NER labels:

Nested labels for NER

Is it possible to create a nested label structure for NER? For example, I have a new entity - DRUG, but also there are several subtypes: ‘antidepressants’, ‘sedative’, ‘cardiac’ etc. Something like: drug_list.jsonl -> {“label”:“DRUG”,“subtype”:“antidepressant”, “pattern”:[{“lower”:“citalopram”}]} {“label”:“DRUG”, “subtype”:“sedative”, “pattern”:[{“lower”:“clonazepam”}]} Or any other solution how to have access to entities and their subcategories. Thanks.

• Last, it's important to know that Prodigy wasn't designed to create "labels-on-the-fly" and that label sets are fixed for each annotation session. You can definitely add new labels in between annotation sessions, but I want to make sure you understand that Prodigy isn't designed to do this within annotation session.

NER - Add labels on the fly

Hi! Prodigy expects you to define the label scheme when you start the annotation process, and if your goal is to collect annotations for machine learning, you typically do not want the annotator to be able to decide about your label scheme and enter labels manually. The presence and absence of a given label is very important and will have a big impact on your entire model. Also, if a new label is introduced later, this can potentially invalidate previous annotations and you'll end up with incon…

Be sure to continue searching on Prodigy support. There may be several other posts -- I found these after a quick search.

I hope this helps and let us know if you have further questions!

[Vishal112]

I have new categories to train and don't want to train the model from scratch, currently I am testing my model in local environment and the data that I have annotated is very large hence I want to skip this step and want to add new lables for new categories I received

[ryanwesslen]

hi @Vishal112!

@Vishal112

the data that I have annotated is very large hence I want to skip this step and want to add new lables for new categories I received

If you decide to retrain (i.e., don't start with a blank:en but a previously trained model), be aware, you'll likely run into problems of catastrophic forgetting. There's not an easy solution to this except being aware that if you retrain with an imbalance of examples (e.g., only retraining on new NER entities), your model may forget the old entities.

New entity model ruins other entities

Sorry about the late reply! I think what you’re experiencing might be whats often referred to as the “catastrophic forgetting problem”. As your model is learning about the new entity type, it’s “forgetting” what it has previously learned. In your example, this is pretty significant – but it might be because you’ve trained a completely new entity, so the only data the model is updating on is examples labelled TECH and none of the other entity types. Because the model is never “reminded” about the…

[Vishal112]

no this is not my concern @ines

consider my scenario
I have 2000 email data with subject(category/Lable) I have annotated the data manually ner.manual receipe
then I have trained the model and I am using it

now I got another 1000 email data with different lables or categories now I dont want to annotate that data which I have annotated previously so what I supposed to do now

pls guide

[Vishal112]

one more thing pls address this issue too why I am getting all label together

[ryanwesslen]

Hi @Vishal112,

@Vishal112

consider my scenario
I have 2000 email data with subject(category/Lable) I have annotated the data manually ner.manual receipe
then I have trained the model and I am using it

now I got another 1000 email data with different lables or categories now I dont want to annotate that data which I have annotated previously so what I supposed to do now

If your new annotation round has different labels, why are you trying to use the model you previously created? Why not create two separate models? Is there overlap in the labels? If so in what way?

@Vishal112

one more thing pls address this issue too why I am getting all label together

I suspect there's an issue with your patterns. Can you provide what your patterns look like? That span is hitting your 3 match pattern (see bottom right) -- you likely are having issues with that pattern.

@Vishal112

Is your ultimate goal an intent model for chats?

If so, why not use a text classification UI instead of a NER for your intent annotations?

Most intent-models do have an accompanying NER model but those NER provide context/additional to help take an action on the intent. You seem to be using an NER for intent (i.e., whether this is account-related, credit-card-related, etc.).

[Vishal112]

No data is same of email bodies but categories are different