Hi there,
I am currently using the entity ruler with ner.make-gold and it's working really well. I tried to use the same model with ner.batch-train and it didn't work - can the entity ruler be used with batch train?
ner.batch-train takes examples and trains a model. The entity ruler is a rule-based component that adds entity annotations based on rules. So they're kinda different things, and the entity ruler is not going to be relevant during training, because the model will be updated with the annotations you provide in the dataset.
So if the results you get after training aren't great, it's related to the data you've collected and are training on.
Okay, I see. Thanks. But that also means that if I save the model, and add it after the entity ruler in a pipeline in spacy, that it may perform better than it does when I train it in prodigy?
If I were to train the model in Spacy, would it use the patterns from the entity ruler?
No, the patterns would never be used during training, because the entity ruler component only writes to the doc.ents at runtime. You can use it to create labelled data for training, though.
Yes, the entity ruler can be used to add entity annotations based on rules at runtime. If it runs before the statistical NER component, the model will "predict around" the entities already set. This can mean that the end result is better. It's a nice trick to influence the model's predictions at runtime, without actually changing anything about the model weights or the model itself.