Advice on training NER models with new entities

[ines (Ines Montani)]

@EFgit

Am I wrong that the purpose of training your own vectors/model using the terms.train-vectors recipe is to create terminology lists and patterns from vectors( terms.teach and terms.to-patterns ) that later you can use in ner.teach recipe?

• If you have a list of terms and patterns to use then such a step would be unnecessary?

Yes, that's correct!

(One quick note on vectors: If you do end up with good vectors for your domain, using them in the base model can sometimes improve accuracy. If you're training a spaCy model and vectors are available in the model, they'll be used during training.)

@EFgit

Based on the Cookbook section Creating gold-standard annotations to annotate from scratch entities that are not present in the model(which is my case) we need to use ner.manual recipe instead of ner.make-gold ?

Yes, that's correct. ner.make-gold can only pre-highlight entities that are predicted by the model, so this only works if the model already knows the entity type. If some of your entitiy types are already present in the model and others aren't, you could also combine the two recipes: start by annotating the existing labels with ner.make-gold, export the data, load it into ner.manual and add the new labels on top. How you do that depends on what's most efficient for your use case.

@EFgit

When you create gold-standard annotation using ( ner.make-gold or ner.manual ) should we do it on the same dataset that we use ner.teach and with the same number of annotations in order to use it as an evaluation set?

Training and evaluation examples should ideally be drawn from the same data source, yes. The examples should also be representative of what your model will see at runtime – for example, if you're processing short paragraphs at runtime, you also want to evaluate the model on short paragraphs (and not, say, short sentences only). Also double-check that there's no overlap between the training and evaluation examples – even single examples can often lead to pretty distorted results. 20-50% of the number of training examples is usually a good amount – if you have under a thousand examples for evaluation, you might have to take the evaluation results with a grain of salt.