Extending SpaCy models with Prodigy to use detect new entity types

[ryanwesslen]

hi @rory-hurley-gds!

Thanks for the background and details on the experiment.

I also noticed you posted on spaCy too.

Extending training of an existing SpaCy model for NER · explosion/spaCy ·...

I would agree with @rmitsch's point:

we strongly recommend training your own NER model from scratch if your target labels change. One reason for that is fine-tuning may lead to catastrophic forgetting of previously learned labels. It's possible to use spaCy's rehearsal functionality to improve model stability, but retraining from scratch is the go-to approach for this as of now.

Have you seen our NER flowchart? We tried to review a similar circumstance of considering training new entity types. A few months ago, we updated the flowchart with relevant posts or documentation.

For example, in that flowchart we recommend if you're trying to add more than three new entities, you're better off to train from scratch.

@rory-hurley-gds

Surprisingly, what we have noticed is that performance is not really good for some of the "original spacy" NER entity types, including GPE, ORG, LOC, FAC.

As Raphael mentioned, you may be dealing with catastrophic forgetting.

Adding new named entities to existing model

Hi @Pragma_Tom, welcome to Prodigy! This is often the case of "catastrophic forgetting" and becomes apparent when some new entities are added to an existing model. Yes. The best practice is to do "pseudo-rehearsal" i.e., to use the original model to label examples and mix them through your fine-tuning updates. As for other strategies, you can check the following threads: Looks like a new trained model has forgotten the old entities New entity model ruins other entities Mixing in gold data t…

I like that post because it has several ideas of ways you can overcome.

@rory-hurley-gds

Is it possible to extend fine tuning of SpaCy models to detect new entity types whilst utilising existing SpaCy entity types?

If you do want to fine-tune, I'd recommend this post from Matt:

https://support.prodi.gy/t/work-flow-for-extending-an-ner

Hope this helps!