# Extending existing entity type with patterns **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner, solved, best-practices\ **Created:** [June 27, 2018, 10:00am UTC](https://support.prodi.gy/t/extending-existing-entity-type-with-patterns/651 "2018-06-27T10:00:24Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![jan-bogaerts](https://avatars.discourse-cdn.com/v4/letter/j/77aa72/32.png) [@jan-bogaerts](https://support.prodi.gy/u/jan-bogaerts)\ **Post date:** [June 27, 2018, 10:00am UTC](https://support.prodi.gy/t/extending-existing-entity-type-with-patterns/651/1 "2018-06-27T10:00:24Z")
Hi, I’m trying to let the system recognize names such as Dr xxx or Mr. xxxx. So I made some manual patterns like this: ``` {"label": "PERSON", "pattern" :[{"LOWER": "dr"}, {"IS_PUNCT": true, "OP": "?" }, {}]} {"label": "PERSON", "pattern" :[{"LOWER": "mr"}, {"IS_PUNCT": true, "OP": "?" }, {}]} ``` and used a command like so: ``` prodigy ner.teach dr_mr_names en_core_web_lg "dr_names.txt" --label PERSON --patterns person_patterns.jsonl --loader txt ``` where the dr\_names.txt file simply contains the text from this site: [http://www.u.arizona.edu/~stoddard/doctor.htm](http://www.u.arizona.edu/~stoddard/doctor.htm), producing a file like so: ``` Dr Klotz (Clots); Dr Wax|Ear, Nose and Throat Dr Pulse, Dr Borer; Dr Cure, Dr Gore Emergency Medicine ``` When I run the recipe, the system suggests names alright, but never with the ‘dr’ also selected. What am I doing wrong? Thanks. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [June 27, 2018, 10:11am UTC](https://support.prodi.gy/t/extending-existing-entity-type-with-patterns/651/2 "2018-06-27T10:11:10Z")
> [@jan-bogaerts](#): > > When I run the recipe, the system suggests names alright, but never with the ‘dr’ also selected. When you look at the meta data displayed in the bottom right corner, does it include a "Pattern" entry? If not, those examples are actually the model's predictions. If you use `ner.teach` with patterns, Prodigy will mix both pattern results and the model's suggestions. Since the model already knows a lot about the label `PERSON`, it starts suggesting stuff right away. And the default NER annotation scheme of the `PERSON` label always excludes the title. If you reject those examples, do you eventually see the pattern matches? Or do they never show up? I just entered your pattern in our [interactive `Matcher` demo](https://explosion.ai/demos/matcher?text=Dr%20Pulse%2C%20Dr%20Borer%3B%20Dr%20Cure%2C%20Dr%20Gore%20%20%20Emergency%20Medicine&model=en_core_web_sm&pattern=%5B%7B%22id%22%3A0%2C%22attrs%22%3A%5B%7B%22name%22%3A%22LOWER%22%2C%22value%22%3A%22dr%22%7D%5D%7D%2C%7B%22id%22%3A1%2C%22attrs%22%3A%5B%7B%22name%22%3A%22IS_PUNCT%22%2C%22value%22%3Atrue%7D%2C%7B%22name%22%3A%22OP%22%2C%22value%22%3A%22%3F%22%7D%5D%7D%2C%7B%22id%22%3A2%2C%22attrs%22%3A%5B%7B%22name%22%3A%22IS_ALPHA%22%2C%22value%22%3Atrue%7D%5D%7D%5D) and it looks alright (ignore the `IS_ALPHA` on the last token – I had to add this because the demo doesn't yet support the empty dict for "any token"). In fact, I think the `{"IS_PUNCT": true, "OP": "?" }` token is even redundant, since spaCy's tokenizer treats "Dr." as an exception and won't actually split it. So instead, you might want to add a second pattern with `{"LOWER": "dr."}`. ---
**Author:** ![jan-bogaerts](https://avatars.discourse-cdn.com/v4/letter/j/77aa72/32.png) [@jan-bogaerts](https://support.prodi.gy/u/jan-bogaerts)\ **Post date:** [June 27, 2018, 11:50am UTC](https://support.prodi.gy/t/extending-existing-entity-type-with-patterns/651/3 "2018-06-27T11:50:58Z")
I changed the patterns a little, first like: ``` {"label": "PERSON", "pattern" :[{"LOWER": "dr"}, {}]} {"label": "PERSON", "pattern" :[{"LOWER": "dr."}, {}]} ``` next like: ``` {"label": "PERSON", "pattern" :[{"LOWER": "dr"}, {}]} {"label": "PERSON", "pattern" :[{"LOWER": "dr."}, {}]} {"label": "PERSON", "pattern" :[{"LOWER": "dr"}]} ``` I ran through the entire set until it said ‘No tasks available’ (wasn’t that big). For the first set of pattherns, It never mentioned anything about ‘pattern’ in the lower right corner, but during the second run with the alternative set of patterns, after a while, it did start to say 'pattern. Sometimes correctly, sometimes only for the word ‘dr’. There were also a number of correct suggestions before it started saying ‘pattern’ (even in the first set of patterns). During the second run, there were more correct suggestions. Am I correct in presuming that it is slowly starting to learn how I would like it to pick persons? ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [June 27, 2018, 12:50pm UTC](https://support.prodi.gy/t/extending-existing-entity-type-with-patterns/651/4 "2018-06-27T12:50:32Z")
> [@jan-bogaerts](#): > > Am I correct in presuming that it is slowly starting to learn how I would like it to pick persons? Yes, that makes sense! The `en_core_web_lg` model already has a pretty solid concept of what a `PERSON` is, so it'll take a while to really move it towards your definition of a person. Depending on what you're trying to do, you might want to consider a combination of the statistical model and a rule-based approach: after all, the model is already pretty good at finding persons, and you can easily improve it even more using Prodigy. The only difference is that it doesn't include the titles like "Dr.". However, if you use spaCy, you'll always have a reference to the tokens surrounding the entitiy. So you could write a custom pipeline component that runs after the regular entity recognizer, checks if the previous token is a title and if so, replaces the entity with a new span that includes the title. Here's a minimal proof of concept: ```python import spacy from spacy.tokens import Span nlp = spacy.load('en_core_web_sm') # or any other model doc = nlp(u"Dr Pulse, Dr Borer; Dr Cure, Dr Gore") # test regular entities print([(ent.text, ent.label_) for ent in doc.ents]) # [('Dr Cure', 'PERSON'), ('Gore', 'PERSON')] def add_titles_to_entities(doc): # this component takes a doc, merges the entities and returns it titles = ['dr', 'mr', 'dr.', 'mr.'] # etc. new_ents = [] # collect the new updated entities for the doc here for ent in doc.ents: if ent.start == 0 or ent.label_ != 'PERSON': # there's no token before the entity or it's not a person new_ents.append(ent) # keep it and exit early continue prev_token = doc[ent.start - 1] if prev_token.lower_ in titles: # previous token text is in list of titles, so create new entity with title ent_with_title = Span(doc, ent.start - 1, ent.end, label=ent.label) new_ents.append(ent_with_title) else: # keep regular entitiy new_ents.append(ent) doc.ents = new_ents # overwrite the entities with the new list return doc # add component to the pipeline – make sure to add it last or after='ner' nlp.add_pipe(add_titles_to_entities, last=True) doc = nlp(u"Dr Pulse, Dr Borer; Dr Cure, Dr Gore") # process text again print([(ent.text, ent.label_) for ent in doc.ents]) # [('Dr Cure', 'PERSON'), ('Dr Gore', 'PERSON')] ``` In this example, you can also see that there's still room for improvement: "Borer" isn't recognised as a person at all. But it's usually _a lot_ easier to improve the existing `PERSON` category and just teach the model to recognise _more_ persons, than it is to change its entire definition and policy, learned from a corpus of over 2 million words. > [@jan-bogaerts](#): > > Sometimes correctly, sometimes only for the word ‘dr’. This was likely a result of this pattern: ```json {"label": "PERSON", "pattern" :[{"LOWER": "dr"}]} ``` ---
**Author:** ![jan-bogaerts](https://avatars.discourse-cdn.com/v4/letter/j/77aa72/32.png) [@jan-bogaerts](https://support.prodi.gy/u/jan-bogaerts)\ **Post date:** [June 27, 2018, 1:50pm UTC](https://support.prodi.gy/t/extending-existing-entity-type-with-patterns/651/5 "2018-06-27T13:50:26Z")
ok. many thanks. I’ll keep these tips in mind (am prepping to write an anonymization tool)