ner.manual pattern file

[dsr2021]

I am using a regex pattern file to assist prodigy in finding patterns for MONEY using regex using ner.manual and en_core_web_lg and a labels file that includes DATE, LOC, GPE, and more. Prodigy correctly identifies all the patterns that I have set for MONEY. It doesn't seem to annotate anything else! Does adding a pattern file stop prodigy from identifying DATE, LOC, GPE? I was under the impression that a patterns file was a a helper to improve prediction and not a replacement for a complete set of recognizing rules. What do I need to do to allow the usual recognition of entities, and create patterns to improve recognition?

[ines (Ines Montani)]

Hi! The ner.manual workflow is fully manual and won't use the model for its predictions (only for tokenization). The main use case of patterns here is to help you pre-label common instances so you don't have to do everything full from scratch.

The ner.correct workflow will stream in the model's predictions and will let you correct them manually. However, it doesn't have an option for patterns because that'd introduce a slightly tricky question about how to deal with conflicts and overlaps, which can often happen. I explain this in more detail in this thread:

@ines

This is a bit more difficult and introduces the problem of how matches vs. predictions should be handled, and which to prefer in case there are overlaps. For NER uses cases, you could default to showing either the prediction or pattern match if they disagree – although, it's often useful to see both, but you still want to make sure that your final data ends up with only one version. And while the span categorizer can predict overlapping spans, you'd often still want to pick one span that's most consistent. For instance, the model may predict a "the" + noun phrase, while your pattern describes only the noun phrase. In that case, you want to make sure that your final data only ends up with one of them, not both. The "comparing annotations" workflow described in this issue goes in a similar direction, and it's definitely something you could implement in a custom recipe: Recipe for comparing NER model and manual annotation - #3 by haishao

One option could be to use ner.manual with patterns for your new categories, and ner.correct with the existing model for all others. When you train your model, Prodigy will automatically merge all annotations on the same text, so it's fine if you have the same example annotated twice with different labels.

Alternatively, you could also implement a small variation of ner.manual that also includes the predictions – you just need to make sure that the data you send out doesn't include any overlaps. You could just filter the spans using spaCy's filter_spans utility and prefer whatever comes first if there's a conflict. Alternatively, you could also decide to prefer pattern matches over predictions, or vice versa. Or you could decide this on a per-label basis – ultimately, this depends on the data and what types of conflicts are most common.