# ner.match error with exact string patterns

**URL:** <https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588>\
**Category:** Uncategorized\
**Tags:** enhancement, usage, ner, done\
**Created:** [June 7, 2018, 12:18pm UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588 "2018-06-07T12:18:58Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![klizter](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/klizter/32/86_2.png) [@klizter](https://support.prodi.gy/u/klizter)\
**Post date:** [June 7, 2018, 12:18pm UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588/1 "2018-06-07T12:18:58Z")

</div>

Hey,

I'm having problems using the `ner.match` recipe for my project. I want to bootstrap a new entity type, so I made a JSONL file with a few exact strings to matches:

```python
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx807"}]}
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx808"}]}
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx809"}]}
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx810"}]}
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx811"}]}
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx812"}]}
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx813"}]}
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx814"}]}
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx815"}]}
{"label": "MASTER_EQUIPMENT", "pattern": [{"lower": "18mx816"}]}

```

However, when i try to run `ner.match` with the patterns file:

```python
→ prodigy ner.match maintenance_reports_annotations en_core_web_lg cleaned_events.jsonl --patterns ./patterns/master_equipment_patterns.jsonl./patterns/master_equipment_patterns.jsonl
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    " __main__", mod_spec)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/prodigy/ __main__.py", line 254, in <module>
    controller = recipe(*args, use_plac=True)
  File "cython_src/prodigy/core.pyx", line 152, in prodigy.core.recipe.recipe_decorator.recipe_proxy
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/plac_core.py", line 328, in call
    cmd, result = parser.consume(arglist)
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/plac_core.py", line 207, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/prodigy/recipes/ner.py", line 50, in match
    model.update(existing)
  File "cython_src/prodigy/models/matcher.pyx", line 109, in prodigy.models.matcher.PatternMatcher.update
KeyError: 'spans'
(env)

```

Am I doing something wrong? I guess there has to be something with the patterns file?

The documentation says:

> ..patterns file can include _exact strings_, regular expressions, or token patterns for use with spaCy's Matcher class....

Thanks in advance! 🙃

`Edit: ` I just tested adding some random words with `terms.teach` and then use the `terms.to-patterns`. However, I'm getting the same error with the new patterns file.

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [June 7, 2018, 1:19pm UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588/2 "2018-06-07T13:19:28Z")

</div>

I just had a look at the recipe and it seems like the error occurs when the existing annotations from the dataset are added to the pattern matcher model. Is it possible that your set `maintenance_reports_annotations` includes other types of annotations that are not NER annotations? Or something without a span? And does the error go away if you use an empty dataset? (You can also set the environment variable `PRODIGY_LOGGING=basic` to see what’s going on behind the scenes.)

Btw, It’s probably unideal that `ner.match` updates the matcher and resumes by default – it’s a nice feature, but we should probably add it behind a flag.

---

<div class="post-metadata">

**Author:** ![klizter](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/klizter/32/86_2.png) [@klizter](https://support.prodi.gy/u/klizter)\
**Post date:** [June 7, 2018, 2:36pm UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588/3 "2018-06-07T14:36:21Z")

</div>

> [@ines](#):
>
> Is it possible that your set `maintenance_reports_annotations` includes other types of annotations that are not NER annotations? Or something without a span?

Yes, that was it! Thank you! I'm not sure how they got there 🤔

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [June 7, 2018, 3:35pm UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588/4 "2018-06-07T15:35:04Z")

</div>

Thanks for updating – glad it worked! (I’ve moved this functionality behind a `--resume` flag btw, which is also more consistent with the other recipes.)

---

<div class="post-metadata">

**Author:** ![klizter](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/klizter/32/86_2.png) [@klizter](https://support.prodi.gy/u/klizter)\
**Post date:** [June 10, 2018, 11:23am UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588/5 "2018-06-10T11:23:08Z")

</div>

I’m running the latest version of prodigy, and keep getting this error when i try to use the patterns file. My patterns file has exactly 6083 entries, which seems to fit with error.

```python
→ prodigy ner.match reports_annotations en_core_web_sm cleaned_events.jsonl --patterns ./patterns/el_equip_patterns.jsonl
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    " __main__", mod_spec)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/prodigy/ __main__.py", line 259, in <module>
    controller = recipe(*args, use_plac=True)
  File "cython_src/prodigy/core.pyx", line 167, in prodigy.core.recipe.recipe_decorator.recipe_proxy
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/plac_core.py", line 328, in call
    cmd, result = parser.consume(arglist)
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/plac_core.py", line 207, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/prodigy/recipes/ner.py", line 54, in match
    model.update(existing)
  File "cython_src/prodigy/models/matcher.pyx", line 112, in prodigy.models.matcher.PatternMatcher.update
IndexError: index 7389 is out of bounds for axis 0 with size 6083

```

Is this connected with the `--resume` flag? Seems like it assumes there are many more entries - maybe because i just ran the recipe with a much bigger patterns file? What can i do?

Edit: Same thing with another patterns file, however different index.

```python
→ prodigy ner.match reports_annotations en_core_web_sm cleaned_events.jsonl --patterns ./patterns/fire_and_gas_patterns.jsonl
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    " __main__", mod_spec)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/prodigy/ __main__.py", line 259, in <module>
    controller = recipe(*args, use_plac=True)
  File "cython_src/prodigy/core.pyx", line 167, in prodigy.core.recipe.recipe_decorator.recipe_proxy
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/plac_core.py", line 328, in call
    cmd, result = parser.consume(arglist)
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/plac_core.py", line 207, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/Users/ocselvig/Code/master_thesis_ner/env/lib/python3.6/site-packages/prodigy/recipes/ner.py", line 54, in match
    model.update(existing)
  File "cython_src/prodigy/models/matcher.pyx", line 112, in prodigy.models.matcher.PatternMatcher.update
IndexError: index 2643 is out of bounds for axis 0 with size 1558

```

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [June 10, 2018, 11:44am UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588/6 "2018-06-10T11:44:57Z")

</div>

> [@klizter](#):
>
> Is this connected with the `--resume` flag? Seems like it assumes there are many more entries - maybe because i just ran the recipe with a much bigger patterns file? What can i do?

Thanks for the report! Unfortunately, the `--resume` flag on `ner.match` didn't actually make it into v1.5.0. I squeezed it in last minute but after we'd published the release, it turned out that the build process had only built from the previous commit 😭 Very sorry about that – it'll definitely come in v15.1.

About the error: Yes, I think your analysis is correct. The scores are stored in a numpy array where the first column is the pattern ID (the value of the span's `"pattern"`) and the second is the annotation decision. The array is created with the same number of rows as your patterns file – so if your existing annotations include examples with a _higher_ pattern ID, the matcher fails to update the scores.

This is an edge case we hadn't considered. We could add a simple check for this, but now that I think about it, we probably want a more sophisticated solution for pattern-heavy use cases. For example, we could hash the patterns to generate the IDs (similar to the input and task hashes).

---

<div class="post-metadata">

**Author:** ![klizter](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/klizter/32/86_2.png) [@klizter](https://support.prodi.gy/u/klizter)\
**Post date:** [June 10, 2018, 11:57am UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588/7 "2018-06-10T11:57:04Z")

</div>

Makes sense! Hehe, no worry 😁

So if I get this correctly, the current implementation of prodigy assumes one single patterns file, and not multiple files for different entity types to bootstrap?

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [June 10, 2018, 12:04pm UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588/8 "2018-06-10T12:04:13Z")

</div>

> [@klizter](#):
>
> So if I get this correctly, the current implementation of prodigy assumes one single patterns file, and not multiple files for different entities to bootstrap?

Yes, because all patterns have a `"label"` assigned, you can keep them all in one file. I definitely see the problems in `ner.match`, though – unline `ner.teach` and most other recipes, it doesn't actually support a `--label` argument at the moment, so there's no convenient way to filter the examples you're seeing. This is also something we should add.

In the meantime, you could just copy-paste the `label` argument from `ner.teach` and then add a stream wrapper like this:

```python
def filter_by_label(stream, labels):
    for eg in stream:
        for span in eg.get('spans', []):
            if span['label'] in labels:
                yield eg

```

And then add it last:

```python
stream = (eg for _, eg in model(stream))
stream = filter_by_label(stream, label)

```

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [June 12, 2018, 3:26pm UTC](https://support.prodi.gy/t/ner-match-error-with-exact-string-patterns/588/9 "2018-06-12T15:26:35Z")

</div>

**Quick update:** Just tested the proposed solution and it’s working well so far. So in the next version, patterns will be referenced by a hash based on the `"label"` and `"pattern"` values. The matcher will also respect a predefined `"id"` (to allow users to implement their own pattern ID system, if needed).

If the `--resume` flag is set, the matcher will be updated from the existing annotations and the hashes ensure that the scores for the correct pattern are incremented, even if different pattern files were used between sessions.

The only downside is that it’s difficult to make the change fully backwards-compatible – but if you have the original list of patterns, you could update the pattern IDs by iterating over your annotations, getting the pattern at index `i` and generating its hash using the `get_pattern_id` helper:

```python
from prodigy.models.matcher import get_pattern_id

patterns = [...] # <-- your list of patterns here

new_dataset = []
for eg in dataset: # your dataset
    pattern_index = eg['meta'].get('pattern')
    if pattern_index: # span was created by a pattern
        pattern = patterns[pattern_index]
        pattern_id = get_pattern_id(pattern)
        eg['meta']['pattern'] = pattern_id
        for span in eg.get('spans', []): # also edit the ID in the span
            span['pattern'] = pattern_id
    new_dataset.append(eg)

```

**Edit:** Just released `v1.5.1`, which includes this update! 🎉