I am using a regex pattern file to assist prodigy in finding patterns for MONEY using regex using ner.manual and en_core_web_lg and a labels file that includes DATE, LOC, GPE, and more. Prodigy correctly identifies all the patterns that I have set for MONEY. It doesn't seem to annotate anything else! Does adding a pattern file stop prodigy from identifying DATE, LOC, GPE? I was under the impression that a patterns file was a a helper to improve prediction and not a replacement for a complete set of recognizing rules. What do I need to do to allow the usual recognition of entities, and create patterns to improve recognition?
ner.manual workflow is fully manual and won't use the model for its predictions (only for tokenization). The main use case of patterns here is to help you pre-label common instances so you don't have to do everything full from scratch.
ner.correct workflow will stream in the model's predictions and will let you correct them manually. However, it doesn't have an option for patterns because that'd introduce a slightly tricky question about how to deal with conflicts and overlaps, which can often happen. I explain this in more detail in this thread:
One option could be to use
ner.manual with patterns for your new categories, and
ner.correct with the existing model for all others. When you train your model, Prodigy will automatically merge all annotations on the same text, so it's fine if you have the same example annotated twice with different labels.
Alternatively, you could also implement a small variation of
ner.manual that also includes the predictions – you just need to make sure that the data you send out doesn't include any overlaps. You could just filter the spans using spaCy's
filter_spans utility and prefer whatever comes first if there's a conflict. Alternatively, you could also decide to prefer pattern matches over predictions, or vice versa. Or you could decide this on a per-label basis – ultimately, this depends on the data and what types of conflicts are most common.