I am having a pattern.jsonl file containing around 13000 lexicons in below manner:
{"label": "PERSON", "pattern": "mariyam"}
{"label": "COUNTRY", "pattern": "uae"}
{"label": "PERSON", "pattern": "ali shah"}
{"label": "PERSON", "pattern": "waseem"}
{"label": "PERSON", "pattern": "sheikh rasheed"}
{"label": "PERSON", "pattern": "arshad"}
{"label": "PERSON", "pattern": "haroon rasheed"}
{"label": "PERSON", "pattern": "khawar ghumman"}
...
Can i train these into prodigy using ner.batch-train? or any other way? as these are just lexicons not sentneces so i want model to just rememebr them, using them in entity ruler is causing me problem in results due to overlapping, so I want them to train it into the model
Hi! Named entity recognition is the task of predicting spans in context, so you also need to train your model on examples in context. You couldn't just update it with single phrases. But you can use Prodigy's ner.manual with --patterns to use your patterns to pre-highlight entities for you. See here for details and examples: https://prodi.gy/docs/named-entity-recognition#manual-patterns
You can then use the resulting annotations and train your model with them. You can still combine the model with rules – just make sure your components don't try to set overlapping entities. The EntityRuler should handle this automatically. If you're using custom components, you can use spaCy's filter_spans utility.